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REPORT  INTRODUCTION 

Deloitte  is  under  contract  W81XWH-07-C-01 15  Department  of  Combat  Medic  Training  - 
Technology  Enhancement  to  the  U.S.  Army  Medical  Research  &  Materiel  Command's 
(USAMRMC)  Telemedicine  &  Advanced  Technology  Research  Center  (TATRC).  During  the 
contract  period  Deloitte  was  purchased  by  Deloitte.  There  were  no  project  impacts  due  to  this 
action. 

The  Department  of  Combat  Medicine  Training  Technology  Enhancement  (DCMT)  team  is 
tasked  with  conducting  the  research,  design,  and  analysis  of  a  technical  solution  to  digitize  and 
integrate  forms-based  testing  and  training  documents  into  the  Army’s  68 W  Training 
Program).  Specific  areas  of  interest  were  the  digital  conversion  of  paper  to  electronic  forms,  and 
two-way  secure  wireless  synchronization  of  the  tablets  and  central  servers  managing  the  data. 
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1  Executive  Summary 

1.1  Approach 

Deloitte  is  under  contract  W81XWH-07-C-0115  Department  of  Combat  Medic  Training  - 
Technology  Enhancement  to  the  U.S.  Army  Medical  Research  &  Materiel  Command's 
(USAMRMC)  Telemedicine  &  Advanced  Technology  Research  Center  (TATRC).  The  DCMT 
project  team  created  this  Concept  of  Operations  (CONOPS)  document  to  details  the  entire 
project’s  research,  design,  testing  and  pilot  efforts  for  integrating  an  automated  testing  solution 
into  the  training  program  for  combat  medics. 


1.2  Exemption  from  Research  Protocol 

This  project  was  determined  to  be  exempt  from  IRB  protocol  per  Appendix 


1.3  What  this  report  says: 

Section  1  -  Executive  Summary: 

(this  section) 

Section  2  -  Introduction: 

This  section  of  the  CONOPS  provides  a  high-level  overview  of  the  project  including  problem 
statements  and  setting. 

The  Department  of  Combat  Medicine  Training  Technology  Enhancement  (DCMT)  team  is 
tasked  with  conducting  the  research,  design,  and  analysis  of  a  technical  solution  to  digitize  and 
integrate  forms-based  testing  and  training  documents  into  the  Army’s  68 W  Training 
Program).  The  specific  area  of  interest  was  the  digital  conversion  of  paper  to  electronic  forms, 
and  two-way  secure  wireless  synchronization  of  the  tablets  and  central  servers  managing  the 
data. 

Section  3  -  Current  State: 

This  section  provides  detail  of  the  current  process  including  the  manual  and  automated  system(s) 
in  use,  description  of  environment,  definitions,  modes  of  operation,  user  classes  and  roles,  and  a 
description  of  the  current  support  environment.  Analysis  of  these  current  process  and  system 
details  is  described  for  the  purpose  of  determining  future  alternatives. 

The  current  process  (at  initiation  of  the  project)  was  a  tedious,  slow  manual  process  involving 
paper-based  checklists  filled  out  by  an  instructor  as  each  student  performs  lifesaving  procedures 
in  a  number  of  clinical  scenarios.  The  completed  checklist  was  hand-delivered  by  runners  to  a 
group  of  evaluators  in  a  separate  location,  where  the  exam  was  scored.  There  were  several 
disadvantages  identified  to  be  addressed  by  the  project  including  delays  in  the  process  and  the 
inability  for  course  managers  to  receive  real-time  reports  of  student  performance.  The  manual 
process  also  required  a  large  group  of  soldiers  to  be  used  as  runners. 

Section  4  -  Future  State  Strategy: 
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This  section  of  the  DCMT  CONOPS  describes  the  case  for  change;  provides  a  recommended 
plan  of  action  based  on  the  findings  of  the  analysis  of  current  system.  The  analysis  of  current 
system  details  is  merged  with  research  into  the  potential  alternative  solutions.  Recommendations 
were  developed  from  this  analysis  and  are  described  in  this  section 

The  specific  area  of  interest  identified  was  the  digital  conversion  of  paper  to  electronic  forms, 
and  two-way  secure  wireless  synchronization  of  a  tablet  device,  and  central  servers  managing  the 
data.  Applied  research  and  analysis  of  the  department’s  business  processes  and  functional 
requirements  will  be  more  thoroughly  understood  for  the  purpose  of  influencing  possible  new 
designs  of  functionality  that  can  be  developed  within  the  68W  program. 

Recommendations  for  a  tablet-based  wireless  solution  were  developed  based  on  the  following 
projected  benefits: 

1 .  Increased  efficiency: 

a.  reduced  paper  use  and  associated  storage  and  management  cost 

b.  free  up  manpower  used  as  runners 

c.  instant  scoring/feedback  including  details  for  corrective  assistance 

d.  decreased  time  required  for  overall  testing  process 

2.  Increased  accuracy 

a.  legible  capture  of  instructor  comments 

b.  validation  of  input  prior  to  submission 

c.  import  of  student  demographics 

3.  Administrative  simplification 

a.  enable  automated  management  of  performance  indicators 

b.  web-enabled  access 

c.  automated  synchronization  of  TEMTPC  and  EMT  Web  Portal 

d.  digital  data  sharing  capability 

4.  Enhanced  data  security 

a.  encryption  during  testing  process  during  transfer  between  instructor  to  evaluator 

b.  encrypted  storage 

c.  encryption  for  transfer 

Section  5  -  Work  Plan: 

This  section  contains  a  timeline  showing  the  milestones  and  deliverables  in  sequential  fashion 
for  both  phase  I  and  phase  II. 

Section  6  -  Concepts  of  the  Pilot  System; 

This  section  displays  and  explains  UML  activity  diagrams  and  use  case  descriptions. 

Section  7  -  Summary  of  the  Pilot  Build  out: 

This  section  provides  details  of  the  components  used  in  the  pilot  project.  The  pilot  was  built 
around  three  components  and  their  installed  software: 

•  Tablet  PC 

•  Laptop  Console 

•  Central  Server 

Section  8  -  Pilot  Operation  and  Test  Results 
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This  section  contains  findings  from  the  pilot’s  operational  use  gathered  during  the  testing  phase 
to  include  a  comprehensive  listing  of  fixes,  changes  and  enhancements  identified  or  requested 
during  interaction  with  staff  on  site  at  the  pilot. 

Section  9  -  Conclusion 


This  section  outlines  the  measured  impact  of  transitioning  from  the  current  state  to  the  new 
concept  piloted  by  providing  the  strategy  map,  listing  of  performance  results  and  general 
comments  and  recommendations  to  address  both  future  operations  in  the  68W  training  program 
and  expanding  the  use  of  this  or  similar  technology  into  other  areas. 

The  performance  results  measured  include  a  significant  reduction  in  paper  documents  being 
stored  and  processed  (paper  forms  for  10,000  exams  provided  per  year),  reduction  of  over  1900 
man-hours  per  year,  and  a  30%  reduction  in  overall  time  for  testing). 


2  Introduction 

The  “68  Whiskey”  (68W  -  Combat  Medic)  military  occupational  specialty  is  the  second-largest 
specialty  in  the  Army,  with  nearly  38,000  medics  spread  across  the  active  and  reserve 
components.  The  Department  of  Combat  Medic  Training  (DCMT)  trains  8,000  new  medics  a 
year,  with  class  sizes  that  stretch  to  nearly  500  students.  A  new  iteration  of  training  starts  every 
two  weeks,  and  at  any  one  time,  as  many  as  2,500  students  are  working  their  way  through  the 
program. 

This  project  was  approved  to  focus  on  improving  a  specific  element  of  the  68 W  training 
program-  the  All  Skills  Practical  Exam  (PE)  required  by  the  National  Registry  of  Emergency 
Medical  Technicians  (NREMT). 


2.1  Document  Overview 

2.1 .1  DCMT  CONOPS  Objective 

The  DCMT  project  team  created  this  Concept  of  Operations  (CONOPS)  document  to  detail  the 
entire  project’s  research,  design,  testing  and  pilot  efforts  for  integrating  an  automated  testing 
solution  into  the  training  program  for  combat  medics.  It  serves  as  the  final  report  for  the  project, 
and  as  a  source  of  recommendations  for  application  of  the  solution  evaluated  and  tested  by  the 
project  team. 

2.1.2  Audience 

•  MHS  end  users 

•  Potential  partners  and  stakeholders  for  follow-on  phases 

•  Developers 

•  Medical  Treatment  Facilities 

•  Training  Site  Cadre' 
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2.1.3  Expected  evolution  of  the  CONOPS 

The  DCMT  CONOPS  will  provide  the  background  and  fundamental  concepts  as  basis  for  next 
steps  for  MHS  implementation/sustainment  activities.  This  is  the  final  report  for  the  project,  and 
as  such,  this  document  is  not  scheduled  for  future  revision  by  the  current  project  team,  however 
it  is  possible  that  changes  could  take  place  during  the  eventual  system  development  process  that 
will  require  subsequent  revisions,  including  potential  change(s)  in  policy  or  overall  system 
requirements. 


2.1.4  Revision  History 


Version 

Description  of  Change 

Author 

Date 

1 

Final  Report 

Raman  Singh 

4/15/2011 

2.2  Project  Overview 

The  DCMT  technology  enhancement  project  was  tasked  with  research,  design,  and  development 
of  a  technical  solution  that  provides  digitized  forms  and  integrated  testing  and  training 
documents  for  the  68W  training  environment.  Solution  design  resulted  in  a  pilot,  which  was 
adopted  by  DCMT  and  continues  to  be  utilized.  Hardware  purchased  was  set  up  in  the 
operational  environment  and  will  remain  on  site  for  use  after  project  termination. 

2.2.1  Problem  Summary 

An  inefficient  manual  paper-based  process  was  being  used  to  evaluate  performance  of  students 
taking  a  practical  exam.  Paper  forms  required  storage  space  and  the  storage  and  retrieval  process 
was  labor  intensive.  Transport  of  the  forms  during  testing  was  also  manual,  introducing  potential 
for  loss  or  breach  of  data  security.  The  process  required  manual  entry  of  demographic  data, 
leaving  room  for  error  in  reading  hand-written  entries,  and  requiring  time  to  validate 
completeness  of  data  entry.  Management  of  performance  indicators  for  the  instructors 
themselves  was  also  manual. 

2.2.2  Approach 

This  project  included: 

•  Performance  of  current  state  analysis  and  industry  research  to  identify  industry  best 
practices  and  trends  to  determine  suitable  solution  set  to  provide  increased  efficiency  for 
the  evaluation  process 

•  A  pilot  of  the  solution  including  design,  development,  testing  and  deployment  of  the 
solution  set  in  an  operational  setting 

•  Compilation  of  recommended  options  for  reaching  the  desired  future  state  including 
recommended  situations  that  could  benefit  from  the  project’s  findings 

•  Action  on  findings  within  scope  and  budget 

•  Final  report 
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2.2.3  DCMT  Technical  Enhancement  Project  Objective 

During  this  study,  the  Deloitte  team,  working  with  the  Department  of  Combat  Medic  Training 
(DCMT),  conducted  research  necessary  to  design  and  develop  a  technical  solution  to  digitize  and 
integrate  forms-based  testing  and  training  documents  into  the  68W  Training  Program. 

This  research  was  focused  on  a  sample  group  of  students  and  instructor(s),  not  the  service 
members  themselves  (This  is  not  a  human  study).  The  population  was  determined  from  students 
and  instructors  who  participate  in  68W  exercises.  The  specific  areas  of  interest  were  the  digital 
conversion  of  paper  to  electronic  forms,  and  two-way  secure  wireless  synchronization  of  the 
tablet,  and  central  servers  managing  the  data.  Applied  research  and  analysis  of  the  department’s 
business  processes  and  functional  requirements  will  be  more  thoroughly  understood  for  the 
purpose  of  influencing  possible  new  designs  of  functionality  that  can  be  developed  within  the 
68W  program  or  in  similar  environments. 

2.2.4  DCMT  Technical  Enhancement  Project  Scope 

The  scope  of  phase  I  of  this  project  was  to  conduct  a  study  to  define  and  test  forms  automation 
and  process  re-design.  Phase  II  expanded  the  initial  findings  to  demonstrate  the  concept  in  a 
pilot  that  includes  providing  hardware  and  training  necessary  to  automate  practical  exams  for  the 
68W  training  program. 

The  project: 

•  Provided  70  programmed  tablets  to  DCMT  to  be  used  by  instructor/evaluators  to  enter 
notes  on  student  progress  as  they  work  through  a  simulated  event 

•  Designed,  loaded  and  tested  customized  forms  software  on  the  tablets  to  emulate  forms 
being  used  for  testing  to  allow  simplified  data  entry 

•  Provided  1  laptop  with  the  console  application  for  use  by  the  course  coordinator  to  view 
and  manage  test  results,  reports,  manage  testing  process  to  include  instructor  performance 
measures 

•  Provided  2  laptops  with  server/storage  capacity  for  automated  synchronized  back-up  and 
additional  data  management  capability 

•  Designed  and  implemented  a  web  portal  allowing  the  data  to  flow  between  elements 
within  the  network  established 

•  Provided  training  to  staff  for  use  of  the  components 

•  Evaluated  the  use  of  the  components  in  a  test  environment,  then  an  operational 
environment 

•  Collected  and  acted  upon  feedback  from  participants  in  the  pilot  to  maximize  the  utility 
of  the  system 
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3  Current  State 

3.1  Background 

The  training  provided  at  DMT  includes  the  All  Skills  Practical  Exam  (PE)  required  by  the 
National  Registry  of  Emergency  Medical  Technicians  (NREMT).  The  EMT  PE  was  established 
to  ensure  each  soldier  at  the  Department  of  Combat  Medic  Training  is  competent  in  thirteen 
separate  skills  trained  from  the  68W  EMT-Basic  curriculum.  The  PE  consists  of  six  skill 
stations;  five  mandatory  stations  and  one  random  skill  station. 

3.2  Description  of  Current  Process 

The  current  process  for  the  PE  was  a  tedious,  slow  manual  process: 

Each  student  was  required  to  perform  lifesaving  procedures  in  a 
number  of  clinical  scenarios.  Using  paper-based  checklists,  an 
instructor  evaluated  each  student's  performance  in  a  scenario.  The 
instructor  begins  the  PE  by  reading  the  scenario  to  each  student,  and 
then  each  student  performs  timed  simulated  lifesaving  procedures  on  a 
fellow  student  who  is  acting  as  the  casualty. 

At  the  end  of  the  scenario  the  instructor  handed  the  graded  checklist  to 
a  runner  who  in  turn  walked  this  checklist  to  a  group  of  evaluators  in  a 
separate  location.  If  the  student  has  passed,  all  is  well.  If  the  student 
has  not  passed  that  station,  he  must  return  later  and  redo  that  particular 
clinical  scenario.  The  student  who  has  failed  the  scenario  is  not 
provided  feedback  as  to  why  he  failed.  He  simply  has  to  repeat  the 
entire  scenario.  The  use  case  diagram  below  documents  the  current  process. 
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3.3  Operational  Environment 

The  EMT  PE  skill  stations  are  remote  from  the  normal  classroom  training  environment  -  at  a 
location  that  simulates  disaster  and  wartime  scenarios.  The  course  evaluators  that  are  monitoring 
student  progress  are  limited  to  recording  progress  based  on  a  check-sheet  with  an  optional  free- 
text  feedback  section  to  document  notes  on  student  behavior  related  to  the  score  given.  Students 
are  not  provided  feedback  during  the  exam. 

4  Future  State  Strategy 

4. 1  Problem  definition 

As  hundreds  of  students  queue  for  the  Practical  Exam,  there  are  many  delays  in  the  process  and 
students  spend  a  great  deal  of  time  just  waiting  for  their  turn  in  the  next  station.  The  manual 
method  did  not  allow  course  managers  to  receive  real-time  reports  of  students’  performance. 
Also,  a  large  group  of  soldiers  were  used  as  runners  to  support  the  event,  taking  them  away  from 
more  meaningful  work  for  an  entire  day. 

4.2  Methodology 

During  this  study,  Deloitte  -  in  support  of  Department  of  Combat  Medic  Training  (DCMT)  - 
researched,  designed  and  developed  a  technical  solution  to  digitize  and  integrate  forms-based 
testing  and  training  documents  into  the  68W  Training  Program.  This  research  focuses  on  the 
process  flow  used  by  instructor(s)  to  evaluate  the  performance  of  a  sample  group  of  students,  not 
the  service  members  themselves.  It  is  not  a  human 
study.  The  population  was  determined  from 
students  and  instructors  who  participated  in  68W 
PEs.  The  specific  area  of  interest  was  the  digital 
conversion  of  paper  to  electronic  forms,  and  two- 
way  secure  wireless  synchronization  of  the  Tablet 
PC,  and  central  servers  managing  the  data. 

After  defining  requirements  and  designing  and 
testing  a  prototype  in  phase  I,  a  pilot  was  conducted 
to  prove  the  concept,  with  the  goal  of  producing  an 
understanding  of  the  department’s  business 
processes  and  functional  requirements  and  to  ensure 
the  design  would  support  the  mission  in  an 
operational  environment. 

This  final  report  and  supporting  document  will 
serve  to  provide  a  record  of  the  results  for  use  in 
improving  the  efficiency  and  accuracy  of  the  6W 
program  and  potentially  apply  lessons  learned  to 
other  programs  as  well.  The  projected  benefits 
included  specific  gains  in  efficiency,  accuracy,  data 
security  and  enhanced  administration  capability. 
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4.3  Prototype  Design 

Based  on  needs  identified  by  DCMT,  a  technical  architecture  was  conceived,  and  requirements 
for  the  solution  design  were  developed. 


EMT  SfTE 


] 


EMT  SERVER 


The  diagram  above  shows  the  functional  design  of  the  technical  architecture.  The  design 
selected  includes: 

1-  Tablet  PCs  used  for  data  entry  and  transmission  using  secure  wireless  transmission 

2-  The  Tablet  EMT  Program  (TEMTP)  -  a  software  program  developed  for  this  testing 
automation  solution  -  resides  on  each  of  the  75  Tablet  PCs  programmed 

3-  EMT  Coordinator  -  an  application  residing  on  a  laptop  station  used  by  the  course  coordinator 

4-  EMT  Server/Web  Portal  -  web  based  portal  providing  access  to  data  stored  on  the  server 

The  instructors  evaluating  students  taking  the  exam  use  Tablet  PCs  with  TEMT  (the  automated 
testing  program)  and  transfer  the  data  using  the  Tablet  PCs  wireless  transmission  capability 
across  a  secure  wireless  closed  network  to  the  EMT  Coordinator.  A  course  coordinator  reviews 
any  test  failures  and  monitors  results  and  performance  of  both  students  and  instructors  using  the 
EMT  Coordinator  processing  center  software,  and  the  EMT  Coordinator  synchs  the  data  with  the 
EMT  Web  Portal  for  storage.  Once  stored  in  the  EMT  Web  Portal,  the  data  can  be  retrieved 
using  an  offline  web  portal  access  protocol,  making  the  data  secure  and  allowing  re-creation  of 
the  form  as  originally  entered  during  the  exam.  The  data  can  also  be  processed  for  reports  and 
analysis  or  for  transmission  to  authorized  requestors  either  through  the  internal  web  portal  or  by 
downloading  encrypted  files  to  any  digital  storage  media  device. 
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4.4  Projected  Benefits 

During  the  assessment  of  the  current  state,  we  identified  several  benefits  to  the  current  processes 
listed  in  the  left  column  and  the  benefit  categories  listed  in  the  top  row  of  the  table  below. 


-  ttml 

Efficiency 

Accuracy 

Simplification 

Security 

Administer  Exam 

Evaluate  Exam 

Management 

Reporting 

Detailed  benefits  are  described  below. 

4.4.1  Increased  efficiency: 

a.  Reduced  paper  use  and  associated  storage  and  management  cost: 

Storage  and  management  of  paper  forms  in  filing  systems  is  a  major  cost  that  is  mitigated  by  use 
of  electronic  systems.  In  the  pilot  the  form  is  never  printed,  is  stored  as  a  digital  file  in  a  central 
database  which  allows  dynamic  recreation  of  the  instructor’s  entry. 

b.  Free  up  manpower  used  as  runners  and  processors: 

Use  of  the  soldiers  as  runners  during  the  testing  is  not  required  for  transfer  of  data  that  is  now  in 
digital  format  from  start  to  finish.  This  allows  the  soldiers  to  be  utilized  for  other  duties. 
Processors  that  took  the  paper  from  the  runners,  scored  exams  and  processed  results  to  be 
returned  by  runner  to  instructors  are  also  reduced  or  eliminated  by  the  automation  of  the  process. 

c.  Instant  scoring/feedback  including  details  for  corrective  assistance: 

Scoring  will  be  tabulated  instantly  upon  submission  by  the  instructor  evaluating  the  student’s 
performance.  The  results  will  display  instantly  for  the  instructor  to  pass  on  the  initial  pass/fail 
status  to  the  student.  In  the  event  of  a  failure,  the  course  training  coordinator  has  the  authority  to 
over-ride  the  results,  and  will  utilize  the  automated  report  including  any  notes  captured  during 
the  evaluation  in  combination  with  other  factors  to  make  the  final  decision.  The  coordinator  can 
utilize  the  “fail  report”  feature  to  generate  a  list  of  students  who  have  failed  a  portion  of  the 
course  in  order  to  perform  a  review  at  any  time  -  allowing  more  efficient  processing  compared  to 
reviewing  paper  forms  brought  by  runners  to  a  collection  point  for  manual  processing.  A 
coordinator  can  provide  the  student  more  comprehensive  rationale  for  their  failure  at  this  point  to 
help  them  prepare  to  re-test. 

d.  Decreased  time  required  for  overall  testing  process: 

The  original  process  of  evaluating  students  included  two  days  -  totaling  approximately  20  hours 
of  time  -  to  run  the  simulation  and  evaluate  two  shifts  of  students  each  day.  The  process 
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efficiencies  gained  from  automating  testing  and  eliminating  some  of  the  wait  times  was  projected 
to  be  significant. 

4.4.2  Increased  accuracy: 

a.  Legible  capture  of  instructor  comments: 

Paper  forms  included  hand-written  entry,  which  is  eliminated  with  the  tablet  based  solution.  The 
instructor  has  an  option  to  enter  notes  in  free  text  with  automatic  conversion  to  digital  text 
(conversion  of  handwriting  to  text)  or  use  pre-scripted  comments  from  a  drop-down  menu. 

b.  Validation  of  input  prior  to  submission: 

The  tablet  will  not  allow  submission  of  an  incomplete  evaluation,  and  requires  the  instructor  to 
validate  the  content  prior  to  submission. 

c.  Import  of  student  demographics: 

The  automated  form’s  population  of  student  demographic  information  for  identification  reduces 
the  potential  for  error  in  filling  out  the  form,  preventing  potential  loss  of  results  due  to 
incomplete  fields. 

4.4.3  Administrative  simplification: 

a.  Enable  automated  management  of  performance  indicators: 

Statistical  analysis  normally  used  to  monitor  the  performance  of  both  students  and  instructors  is 
automated  and  can  be  accessed  using  pre-loaded  report  formats.  This  allows  analysis  of  patterns 
including  pass/fail  rates,  multiple  failures  at  same  site,  relating  the  failures  to  specific  factors  has 
potential  to  improve  the  quality  of  the  training  program.  Reports  are  produced  with  a  click  rather 
than  having  to  transcribe  and  analyze  paper  documents  into  computable  format. 

b.  Web-enabled  access: 

Ability  to  access  various  reports  and  features  of  the  system  using  any  computer  with  web  access 
allows  flexibility  for  application  use. 

c.  Automated  synchronization  of  EMT  Coordinator  and  EMT  Web  Portal: 

As  digital  data  must  be  protected  from  loss  either  by  hardware  failure  or  theft,  The  EMT 
Coordinator  will  automatically  transfer  or  synch  all  records  to  a  central  repository  residing  on  a 
local  secure  web  server  -  the  EMT  Web  Portal.  This  will  not  only  free  up  space  on  the  device  - 
allowing  it  to  work  more  efficiently  -  but  will  provide  a  failsafe  location  to  store  all  records  for 
all  exam  locations.  In  the  event  of  hardware  failure  where  data  would  normally  be  loss,  a 
coordinator  or  leadership  can  easily  retrieve  (i.e.  view  or  print)  that  data  from  the  server  by 
logging  into  a  web  portal.  The  web  service  will  encrypt  and  transfer  the  records  from  TEMTPC 
application  to  EMT  Web  Portal  from  the  click  of  a  button.  Once  all  data  has  been  successfully 
synchronized  EMT  will  notify  the  coordinator  of  its  successful  completion.  This  action  can  be 
instantly  verified  by  login  into  the  EMT  Web  Portal  and  viewing  a  log  of  all  synchronized 
records. 

d.  Digital  data  sharing  capability: 

When  data  requests  are  received,  such  as  requests  from  the  National  Registry,  there  was 
extensive  manual  effort  required  to  prepare  and  share  paper-based  forms.  The  solution  designed 
would  allow  access  to  be  granted  through  EMT  Web  Portal  if  approved,  or  data  can  be  easily 
exported  by  selected  criteria  to  a  portable  medium  (i.e.  CD  or  Tape).  This  feature  can  save 
shipping  costs  as  well.  The  data  stored  will  allow  dynamic  recreation  of  the  exact  format  and 
content  of  the  data  entered  on  the  screen  of  the  tablet  by  the  instructor. 


Deloitte-Federal  Healthcare 


15  of  57 


Final  Report  /  Concept  of  Operations  ( CONOPS ) 


DCMT  TE  W81 XWH-07-C-01 1 5 


4.4.4  Enhanced  data  security: 

It  is  essential  to  maintain  confidentiality  of  each  candidate’s  records.  All  components  will 
provide  128-bit  encryption.  As  data  will  be  transmitted  from  component  to  component  it  will  by 
encrypted  using  a  128-bit  algorithm  to  ensure  privacy  and  prevent  theft.  Encryption  applies  to 
three  states  of  the  data  -  during  the  initialization  process,  when  the  data  is  entered  by  instructors 
and  processed  to  include  providing  results,  storage,  and  transfer  from  storage  to  a  requestor. 

a.  Encryption  during  testing  process  during  transfer  between  instructor  to  evaluator: 

Data  is  encrypted  as  soon  as  it  is  submitted  in  the  tool  for  transmission  during  evaluation. 

b.  Encrypted  storage: 

Once  entered,  the  storage  of  data  includes  encryption  to  ensure  that  data  cannot  be  used  if  stolen 
or  shared  without  approval. 

c.  Encryption  for  transfer: 

The  data  can  be  transferred  via  the  web  portal  or  copying  to  storage  media  -  remaining  encrypted 
for  secure  transfer. 


5  Work  Plan 


The  timeline  below  is  divided  into  three  phases:  the  first  two  were  performed  by  Deloitte  and  the 
last  one  reflects  the  ongoing  field  use  of  the  tools  and  methods  that  were  provided.  It  shows 
major  milestones,  with  expanded  milestones/deliverables  listed  in  the  key  below. 


► 


2007 


2011 


5.1  PHASE  I 


1.  Project  Start 

2.  Define  Project  Scope 

3.  Functional  Requirements  -  What  were  the  requirements  needed? 

4.  System  Requirement  Specifications  -  How  would  the  system  meet  these  needs? 

5.  System  Design 

6.  Prototype  build  and  testing  -  Working  in  an  office  environment,  test  the  devices. 

7.  Validation  of  system  functionality 

8.  Phase  I  end 


5.2  Phase  II 

8.  Project  Start 

9.  System  Build  -  Build  the  applications  and  wireless/server  environment 

10.  Certification  and  Accreditation 
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11.  System  Testing  &  Training 

12.  System  Delivery 

13.  Technical  Support  Plan  &  Documentation 

14.  Final  Report 

15.  Project  End 

Operational  Use 

16.  Implement  Recommendations 

6  Concepts  of  the  Pilot  System 

The  system  transforms  the  current  paper-based  Practical  Exam  into  an  electronic  format  residing 
on  a  Tablet  PC  based  secure  wireless  network.  It  digitizes  the  forms  and  the  flow  of  the  data 
from  the  Tablet  PCs  to  the  testing  coordinator  and  finally  to  the  DCMT  central  servers  using  a 
web  portal  application. 

6. 1  DCMT  Activity  Diagram 

This  system  digitizes  the  testing  forms,  and  helps  to  improve  the  efficiency  of  the  instructional 
learning  and  examination  process  as  depicted  in  the  UML  Activity  Diagram  below. 


DMCT  Activity  Diagram  for  Practical  Examinations 
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6.2  DCMT  Use  Cases 

The  system  supports  the  following  objectives  using  mobile  wireless  environment: 

1 .  Collect  metrics  in  real-time, 

2.  Analyze  the  collected  metrics, 

3.  Provide  feedback  based  on  the  analysis,  and 

4.  Generate  reports  based  on  the  feedback  to  aid  further  improvement  of  the  training  and 
testing  process. 

In  this  research,  we  documented  the  Use  Cases  using  a  standard  object  based  UML  modeling 
technology  IBM  Rhapsody  7,5.  Information  and  models  are  contained  in  Appendix  A  and  B. 

The  Use  Case  models  identify  the  interaction  between  the  System  (DCMT)  and  the  various 
Actors  using  that  system.  Each  use  case  is  then  broken  down  into  a  series  of  system 
requirements. 


These  system  requirements  will  then  be  housed  around  a  three  tiered  set  of  Layers 

6.3  DCMT  Technology  Layers 

The  system  contains  three  layers  of  technology  for  collection,  processing  and  reporting. 

•  Layer  1:  Tablet  PC  Forms  Application  to  collect  and  review  data 

•  Layer  2:  Local  Laptop  Server  Application  to  process  and  analyze  secure  digital  data 

•  Layer  3:  EMT  Web  Portal  and  Central  Database  Server  for  real  time  reporting  and 

verification 
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6.3.1  Layer  1 :  Tablet  PC  Forms  Application 

EMT  Tablet  PC  Application  -  The  EMT  Tablet  PC  application  is  used  by  the  instructors  to 
digitally  capture  data  entered  into  an  electronic  version  of  the  EP  exam.  The  application 
replicates  an  exact  electronic  copy  of  the  13  EP  forms  to  provide  the  comfort  and  recognition  to 
the  instructor  of  the  actual  paper  form.  The  application  provides  validation  controls  so  that 
necessary  information  is  collected  and  entered.  The  electronic  form  will  automatically  display 
the  candidate  information  and  examiner’s  name. 

6.3.2  Layer  2:  Local  Laptop  Server  Application 

This  application  will  be  used  by  the  site  Coordinators  to  manage  instructors  and  candidates  as 
well  as  facilitate  data  gathering,  reporting  and  analysis.  The  data  submitted  by  instructors  at  the 
site  is  stored  on  the  local  laptop  server  and  managed  by  the  application.  The  application  scores 
the  form  and  inserts  data  into  a  secure  database.  This  database  contains  the  forms’  information, 
which  is  used  for  real-time  exam  activity  and  real-time  reporting.  The  application  allows  the 
EMT  coordinator  to  synchronize  data  stored  in  the  local,  central  repository  of  the  EMT  central 
server,  viewed  via  a  secure  web  portal. 


6.3.3  Layer  3:  EMT  Web  Portal  and  Central  Research  Data  Collection 
System 

The  EMT  Coordinators  and  Leadership  use  the  EMT  Web  Portal  application  to  view  and  analyze 
the  data  submitted  by  EMT  sites.  Data  submitted  by  the  EMT  sites  is  synchronized  from  the 
local  EMT  sites  into  the  central  EMT  database  server.  This  database  server  acts  as  the  permanent 
repository  to  store  EMT  related  data.  The  EMT  Web  Portal  application  accesses  the  EMT  central 
database  server  to  generate  reports  and  acts  as  a  data  archiving  tool  as  well.  The  EMT  web  portal 
restricts  access  to  those  individuals  who  are  authorized  to  view  data. 
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6.4  Detailed  Use  Cases  Aligned  with  Requirements 

The  requirements  are  presented  in  the  format  I  Req  X.X  where  X  reflects  the  tag  number 
assigned  to  each  requirement.. 

6.4.1  Practical  Exam  Administration 

This  Use  Case  scope  entails  the  actions  performed  by  the  instructor  or  evaluator  to  document  the 
practical  exam.  It  consists  of  the  two  use  cases  of 

•  Select  Student  and  Administer  the  Exam 

•  Score  the  Exam 

There  are  five  high  level  requirements  that  support  these  two  cases  and  are  shown  in  the  diagram 
below. 


6.4.2  Req  1.3.SELECTION  OF  CANDIDATE  AND  EXAMINATION 

1.3.1  -  The  system  will  fetch  the  list  of  scheduled  candidates  from  the  local  database  every  day. 

1.3.2  -  The  system  will  display  a  list  of  candidates  scheduled  for  that  day  alone  and  nobody  else. 

1.3.3  -  The  evaluator  will  select  a  candidate  from  the  list. 

1.3.4  -  The  system  will  then  display  the  list  of  13  exams  for  the  candidate,  allowing  access  only 
to  those  forms  that  were  not  already  graded  by  the  evaluator  earlier  on  the  same  day. 

1.3.5  -  The  system  will  not  allow  access  to  any  forms  for  candidates  who  have  failed  four  exams 
or  the  same  exam  three  times. 

6.4.3  Req  1.4.SCORING  OF  EXAMS 

1.4.1  -  The  system  will  allow  the  evaluator  to  check  or  uncheck  responses  to  the  questions  based 
on  the  actions  performed  by  the  candidate. 

1.4.2  -  The  system  will  allow  the  evaluator  to  check  or  uncheck  certain  critical  criteria  based  on 
whether  or  not  the  candidate  performed  some  important  actions. 
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6.4.4  Req  1.5.COMMENTS 

1.5.1  The  system  will  allow  the  evaluator  to  note  down  remarks  and  comments  about  the 
candidate’s  performance  and  a  list  of  the  actions  performed  by  the  candidate 

6.4.5  Req  1.7.FORM  ACCESS 

1.7.1  The  system  will  not  allow  a  candidate  to  test  for  any  particular  exam  with  a  particular 
evaluator  more  than  once. 

1.7.2  The  system  will  not  allow  a  candidate  who  has  failed  the  same  test  thrice  or  four  tests  on 
the  whole  to  take  any  further  tests  on  the  same  day. 

6.4.6  Req  1.8.DATA  VALIDATION  AND  FORM  SUBMISSION 

1.8.1  The  system  will  not  allow  an  incomplete  form  (i.e.  One  where  all  fields  have  not  been 
filled)  to  be  submitted. 

1.8.2  The  system  will  accept  a  signature  of  the  evaluator  and  only  then  will  it  let  the  form  to  be 
submitted. 

1.8.3  The  system  will  authenticate  the  evaluator’s  signature  with  the  last  four  digits  of  the 
evaluator’s  Social  Security  Number,  and  only  then  will  the  signature  be  accepted  and  the  form 
allowed  to  be  submitted. 

1.8.4  The  system  will  allow  evaluators  to  make  changes  even  after  the  form  has  been  signed,  but 
the  form  has  to  be  resigned  before  being  submitted. 

1.8.5  The  system  will  insert  all  submitted  records  into  the  local  server’s  database  immediately 
via  a  wireless  connection. 

1.8.6  The  system  will  not  allow  evaluators  to  make  changes  to,  or  view  submitted  forms. 
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6.5  Evaluate  Exams 

The  diagram  below  shows  the  high  level  requirements  that  support  the  evaluation  of  the  exam 
results. 


Layer  1 
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6.5.1  Req  1.6  PASS/FAIL  CRITERIA 

1.6.1  -  The  system  will  consider  a  candidate  who  has  a  score  greater  than  the  minimum  score 
required  to  pass  the  skill  and  no  checked  critical  criteria,  to  have  passed  the  skill. 

1.6.2  -  Any  candidate,  not  meeting  any  of  the  above  stated  requirements,  is  considered  to  have 
failed  the  skill. 

6.5.2  Req  2.3  VIEW  EXAMS 

2.3.1  -  The  application  will  allow  the  coordinator  to  view  submitted  exams  (both  passed  and 
failed)  for  review. 

2.3.2  -  The  application  will  allow  the  coordinator  to  view  the  exams  based  on  groups  and  teams. 

2.3.3  -  The  application  will  allow  the  coordinator  to  view  graded  individual  skills. 

6.5.3  Req  2.4  OVERRIDE 

2.4.1  -  The  application  will  allow  the  coordinator  to  rescore  submitted  exams  should  there  be  a 
need. 

2.4.2  -  The  application  will  allow  the  coordinator  to  submit  the  rescored  exams  only  if  the 
candidate  receives  a  passing  grade  after  rescoring. 
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6.5.4  Req  2.6  MANAGE  APPLICATION  FUNCTIONALITY 

2.6.1  -  The  application  will  allow  the  coordinator  to  delete  submitted  forms  should  there  be  a 
need. 

2.6.2  The  application  will  allow  the  coordinator  to  delete  all  or  particular  synchronized  records. 


6.6  Systems  Administration  Use  Case 


6.6.1  Req  1.1  AUTHENTICATION 

1.1.1  -  The  users  will  be  created  on  the  local  EMT  Console  server  by  the  site  coordinator. 

1.1.2  The  system  will  allow  only  authorized  users  (i.e.  the  evaluators),  to  log  in. 

1.1.3  The  system  will  authenticate  the  users  by  verifying  the  username  and  password  entered  by 

1.1.4  the  user  against  the  ones  set  in  the  EMT  local  database  server. 

1.1.5  The  system  will  lock  the  user  account  if  the  wrong  username  or  password  is  entered  thrice. 

1.1.6  The  system  will  require  the  users  to  reset  their  passwords,  every  90  days. 

6.6.2  Req  1.2  APPLICATION  CONFIGURATION 

1.2.1  The  users  will  be  required  to  configure  the  system,  the  first  time  they  login. 


Deloitte-Federal  Healthcare 


23  of  57 


Final  Report  /  Concept  of  Operations  ( CONOPS ) 


DCMT  TE  W81 XWH-07-C-01 1 5 


1.2.2  They  will  need  to  provide  the  name  or  IP  Address  of  the  local  laptop  server  which  hosts  the 
local  database. 


6.6.3  Req  2.1  AUTHENTICATION 

2.1.1  The  application  will  have  an  administrator  username  and  password,  which  will  be  provided 
only  to  the  site  coordinator. 

2.1.2  The  system  will  lock  the  user  account  if  the  wrong  username  or  password  is  entered  thrice. 

2.1.3  The  system  will  require  the  users  to  reset  their  passwords,  every  90  days. 

6.6.4  Req  2.2  MANAGEMENT  OF  USERS  AND  CANDIDATES 

2.2.1  The  application  will  allow  the  coordinator  to  prepare  the  list  of  scheduled  candidates  for 
the  day,  either  by  adding  them  manually  or  by  importing  from  an  Excel  spreadsheet. 

2.2.3  The  application  will  allow  the  coordinator  to  edit  the  profiles  of  existing  candidates. 

2.2.4  The  application  will  allow  the  coordinator  to  add  evaluators,  create  usernames  and 
passwords  for  them,  and  edit  and  manage  their  profiles. 

2.2.5  The  application  will  allow  the  coordinator  to  add  groups  and  teams  and  create  and  delete 
mappings  between  them. 

2.2.6  The  application  will  allow  the  coordinator  to  assign  candidates  to  particular  groups  and 
teams. 

6.6.5  Req  2.7  SECURE  DATA 

2.7.1  All  data,  instructors’,  candidates’  and  exams’,  will  be  stored  in  a  secure  MS  SQL  Server 
Database. 

2.7.2  All  wireless  data  flow  between  the  tablets  and  the  local  server  will  use  the  DoD  certified 
encryption  software  “AirFortress”. 

2.7.3  All  data  flowing  between  the  local  server  and  the  central  server  will  adhere  to  DoD 
network  policy. 

6.6.6  Req  2.8  DATA  SYNCHRONIZATION 

2.8.1  The  application  will  allow  the  coordinator  to  synchronize  the  records  with  the  central 
database  using  a  web  service. 

2.8.2  The  application  will  authenticate  the  user  before  synchronizing,  by  getting  them  to  enter  a 
username  and  password,  which  was  created  when  they  registered  as  a  user  on  the  web  portal. 

2.8.3  The  application  will  run  the  web  service  over  a  Secure  Socket  Layer  (SSL)  on  port 
443 (https). 

6.6.7  Req  3.1  AUTHENTICATION 

3.1.1  The  web  service  will  use  the  same  username  and  password  that  was  created  by  the  user 
when  he/she  registered  on  the  web  portal. 

3.1.2  This  username  and  password  will  be  required  to  be  provided  when  synchronizing  records 
from  the  local  laptop  server,  which  is  the  only  way  to  access  the  web  service. 
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6.6.8  Req  3.2  DATA  INSERTION 

3.2.1  The  web  service  will  insert  the  data  it  receives  from  the  local  laptop  server  during  the 
synchronization  process,  into  the  central  database. 

3.2.2  The  data  in  the  central  database  will  be  accessed  via  the  web  portal  alone  and  by  registered 
portal  users  alone. 

6.6.9  Req  3.3  DATA  SECURITY 

3.3.1  The  web  service  will  be  run  over  a  Secure  Socket  Layer  (SSL)  on  port  443  (https). 

3.3.2  The  data  in  the  central  database  will  be  accessed  via  the  web  portal  alone  and  by  registered 
portal  users  alone. 

6.7  Web  Portal  Use  Case 


6.7.1  Req  4.1.  REGISTRATION 

4.1.1  The  portal  will  allow  potential  users  to  register  for  accounts  by  providing  their  Name, 
Social  security  Number  and  e-mail  address. 

4.1.2  The  users  will  also  specify  the  kind  of  user  group  they  would  like  to  register  for  like 
Coordinator,  Administrator  or  Evaluator. 

4.1.3  The  administrator  will  then  review  the  details  provided  and  approve  or  disapprove  the 
users. 

4.1.4  The  portal  will  send  out  an  e-mail  to  the  requestors,  informing  them  about  the  status  of 
their  request. 

4.1.5  The  e-mail  will  also  contain  the  username  and  password,  which  will  be  auto-generated  by 
the  portal,  should  the  request  be  accepted. 
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4.1.6  The  portal  will  ask  the  users  to  change  their  password  on  the  first  login  and  later 
periodically. 


6.7.2  Req  4.2  USER  AUTHENTICATION 

4.2.1  The  portal  application  will  verify  the  username,  password  and  account  type,  before  letting 
the  users  log  in  to  the  application. 

4.2.2  The  portal  will  allow  only  three  types  of  users:  Administrators,  Coordinators  and 
Evaluators. 

4.2.3  The  portal  will  allow  the  administrator  to  manage  users  and  their  profiles,  process  requests 
for  accounts  and  nothing  more. 

4.2.4  The  portal  will  allow  coordinators  to  view  all  records,  synchronized  from  their  site  alone. 

4.2.5  The  portal  will  allow  evaluators  to  view  records  submitted  by  them  alone  and  nothing 
more. 

4.2.6  The  system  will  lock  the  user  account  if  the  wrong  username  or  password  is  entered  thrice. 

4.2.7  The  system  will  require  the  users  to  reset  their  password,  every  90  days. 

6.7.3  Req  4.3  REPORTS  AND  RECORDS 

4.3.1  The  portal  will  allow  the  coordinator  to  view  individual  exam-based  reports. 

4.3.2  The  portal  will  allow  the  coordinator  to  view  rescored  exams-based  reports. 

4.3.3  The  portal  will  allow  the  coordinator  to  view  evaluator-based  reports. 

4.3.4  The  portal  will  allow  the  coordinators  to  view  individual  exams  submitted  from  their  site 
and  also  exams  that  were  rescored  by  them. 

4.3.5  The  portal  will  allow  the  evaluators  to  view  exams  and  reports  pertaining  to  exams 
submitted  by  them  alone,  and  evaluator-based  reports  pertaining  to  them  alone. 

4.3.6  The  portal  will  not  allow  the  evaluators  to  view  any  rescored  exam  or  rescored  exams 
report. 

6.7.4  Req  4.4  DATA  ARCHIVING 

4.4.1  The  portal  will  serve  as  a  central  repository  for  all  data  collected  using  the  Local  Laptop 
Server  application  and  the  Tablet  PC  application. 

4.4.2  The  portal  will  allow  for  coordinators  or  leadership  to  retrieve  any  exam,  transfer  exam 
data  to  a  portable  media  or  print  a  replica  of  the  exam. 

6.7.5  Req  4.5  DATA  ACCESS: 

4.5.1  The  portal  will  allow  the  coordinators  to  view  individual  exams  submitted  from  their  site 
and  also  exams  that  were  rescored  by  them. 

4.5.2  The  portal  will  allow  the  evaluators  to  view  exams  and  reports  pertaining  to  exams 
submitted  by  them  alone,  and  evaluator-based  reports  pertaining  to  them  alone. 

4.5.3  The  portal  will  not  allow  the  evaluators  to  view  any  rescored  exam  or  rescored  exams 
report. 
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6.8  Application  and  Data  Security  Requirements 

6.8.1  Req  5.1  SECURE  AUTHENTICATION 

5.1.1  The  portal  will  automatically  generate  the  username  and  password  for  the  users  by  using 
their  first  name,  last  name  and  social  security  number. 

5.1.2  The  portal  will  create  a  password  based  on  the  following  rules: 

5. 1.2.1  Password  will  have  a  minimum  length  of  10  characters 

5. 1.2.2  Password  will  contain  at  least  2  each  of  upper-case  alphabets,  lower-case  alphabets, 
numbers  and  special  characters. 

5.1.3  The  portal  will  request  the  user  to  change  the  password  on  the  first  login  and  again 
periodically. 

5.1.4  The  access  to  Tablet  PC  and  EMT  Console  will  be  managed  by  the  DoD  Network  Policy. 

5.1.5  The  users  for  the  Tablet  PC  and  EMT  Console  applications  will  be  created  on  the  EMT 
Console  server. 

5.1.6  The  same  password  rules  will  apply  to  users  created  for  the  local  EMT  Console  server 
application  and  the  Tablet  PC  application. 

6.8.2  Req  5.2  DATABASE  AUTHENTICATION 

5.2.1  Any  access  to  the  database  will  be  authenticated  by  a  username  and  password  which  will  be 
provided  by  the  application  once  the  user  has  been  authenticated. 

6.8.3  Req  5.3  DATA  TRANSMISSION  SECURITY 

5.3.1  The  data  being  transmitted  by  the  wireless  network  will  be  protected  by  the  DoD  approved 
encryption  software  “AirFortress”,  available  at  the  site  of  deployment. 

5.3.2  The  data  being  transmitted  by  the  wired  network  will  be  protected  by  the  security 
mechanisms  available  on  the  DoD  network. 
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7  Summary  of  the  Pilot  Build  Out 

The  Pilot  project  was  built  around  three  components  and  their  installed  software: 

•  Tablet  PC 

•  Laptop  Console 

•  Central  Server 

7.1  Pilot  Hardware 


7.1.1  TABLET  PC  Specifications 

•Brand:  Fujitsu 

•Model  Number:  Tablet  PC 

•Processor:  INTEL  CORE2  DUO 

Ultra  Low  Voltage  U7600 
•Screen:  12. 1"  XGA  TFT  with 

indoor/outdoor  display  with  wide 
viewing  angles 

40  GB  S-ATA  150, 


•Hard  Drive 
5400  rpm  hard  drive 
•Memory: 
memory  (1  GB  x  1) 

•Main  battery: 

•Other: 

multinationals  56K4  V.90  modem  and 
Gigabit  Ethernet  LAN  Integrated  Intel® 
PRO/Wireless  3945 ABG  Network  Connection 
(Tri-mode  802.11a/b/g)  Integrated  Fingerprint 
Sensor,  embedded  TPM,  dedicated  Smart  Card  slot,  Fujitsu  Security  Application  Panel 


1  GB  DDR2  667  MHz 

Lithium  ion  (9-cell) 
Integrated 


7.1 .2  Local  Laptop  Server  Specifications 


Local  Laptop  Server 
•Brand:  Any 

•Model  Number:  Any 

•Processor:  Intel®  Core™  2  Duo 

processor  (2.4GHz  800MHz  3MBL2) 

•Operating  System:  Windows  XP  Professional 
•Screen:  14.1  SXGA+ TFT 

•Hard  Drive  200  GB  Hard  Disk  Drive, 

5400rpm  with  Disk  Encryption 
•Memory:  2  GB  RAM 

•Battery:  9  cell  Li-Ion  Battery 

•Keyboard:  Keyboard  US  English 

•  Optical  device:  DVD  Recordable  8x  Max  Dual  Layer,  Ultrabay  Slim 

•  Pointing  Device:  UltraNav  (TrackPoint  and  TouchPad) 

•  Expansion  slots:  PC  Card  Slot  &  Smart  Card  Slot 
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Wireless  Card:  802. 1  la/b/g/n  Wi-Fi  wireless  LAN  card 

Power  cord:  Country  Pack  North  America 


Optical  device: 
Pointing  Device: 
Expansion  slots: 
Power  cord: 


7.1.3  Central  Server  Specifications 

Central  Server 
•Brand:  Any 

•Model  Number:  Any 

•Processor:  Intel®  Core™  2  Duo 

processor  T8300  (2.4GHz  800MHz 
3MBL2) 

•Operating  System:  Windows  Server  2003 
•Hard  Drive  200GB  Hard  Disk  Drive 

•Memory:  2  GB  *  2 

DVD  Recordable  8x  Max  Dual  Layer 
UltraNav  (TrackPoint  and  TouchPad) 

N/A 

Country  Pack  North  America 


7.2  Software  Specifications 

7.2.1  TABLET  PC  Software 

•  The  application  will  require  Microsoft  .Net®  Framework  v  3.5. 

7.2.2  Local  Laptop  Server  Software 

•  The  application  will  require  Microsoft  .Net®  Framework  v  3.5. 

•  The  application  will  require  Microsoft  SQL  Server®  2005  with  reporting  services  tools. 

•  The  application  will  require  Microsoft  Internet  Information  Services®  (IIS®)  v  6.0. 

•  The  application  will  require  Microsoft  Web  Services  Enhancements®  (WSE®)  v  3.0. 

•  The  application  will  require  Microsoft  Office®  2003  or  2007  (preferably  2007)  to  be 

installed  on  the  local  laptop  server  and  the  systems  running  the  web  portal  to  be  able  to  export 
reports  to  an  MS  Excel®  spreadsheet. 

7.2.3  Central  Server  Software 

•  The  application  will  require  Microsoft  Internet  Explorer®  (IE®)  v  6.0  or  later. 
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8  Pilot  Operation  and  Test  Results 

Using  the  requirements  from  Section  7&8,  Deloitte  built  a  pilot  DCMT  system.  Many  additions 
to  functionality  were  added  during  this  phase  with  a  compiled  functionality  list  in  the  subsections 
that  follow. 

Pilot  systems  testing  for  compliance  with  the  requirements  were  performed  in  the  laboratory  in 
April  2008.  Work  continued  on  the  project  until  January  2010  where  final  field  testing  was 
conducted  using  actual  subjects.  We  will  describe  the  high  level  testing  results  in  this  section 
with  complete  testing  results  provided  in  Appendix  C. 

8. 1  Preliminary  Testing  Results 

System  preliminary  testing  occurred  over  several  days  in  late  March  2008. 

Second  test  March  2009 
Final  testing  January  2010 

8.1 .1  Evaluator  Tablet  PC  Functionality 

The  table  below  shows  the  eventual  requirements  created,  the  testing  action  and  the  expected 
system  responses. 


Requirements 

Description 

Action 

Expected  Results 

user  enters  his 
credentials  to  login 
into  the  tablet  PC 

user  enters  his/her  correct 
User  Id  and  password 

The  system  authenticates  the  user  based  on 
his/her  credentials  against  the  EMT  database 
and  displays  the  pending  Candidate  list  for  the 
current  set  of  exam  days 

user  enters  incorrect 
credentials 

user  enters  incorrect  User  Id 
and  password 

The  system  prompts  the  user  with  message 
"Please  enter  correct  user  name  and 
password" 

List  of  pending 
candidates  is 
displayed 

Once  the  user  credentials 
are  verified  and  he/she  is 
logged  in,  the  system 
displays  the  list  of  pending 
candidates  who  are 
scheduled  for  the  current  set 
of  exam  dates. 

System  displays  the  scheduled  candidates  for 
the  current  set  of  exam  dates.  This  list  is  sorted 
alphabetically  based  on  the  last  name  of  the 
candidates. 

user  clicks  on  the 
Update  Profile  button 
to  be  able  to  change 
his/her  name,  SSN, 
Password  and  PIN 

The  user  clicks  on  the 

Update  Profile  button 

Another  window  opens  up  with  the  First  and 

Last  names  and  the  Username  and  fields  to 
update  the  password  and  PIN. 

User  changes  name, 
password  and/or  PIN 

The  user  changes  his/her 

First  and/or  Last  name 
and/or  enters  correct  old 
password  and  new 
password  and/or  correct  old 
PIN  and  new  PIN,  which 
have  not  been  used  in  any  of 
the  last  10  password/PIN 
changes,  twice(once  as 

System  displays  a  message 
"Name/Password/PIN  has  been  successfully 
updated" 
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confirmation)  and  clicks  on 
the  Update  Profile  button 

User  enters  incorrect 
old  Password/PIN 

The  user  enters  an  incorrect 
old  Password  and/or  PIN 

System  prompts  the  user  with  message 
"Incorrect  old  password/PIN  entered" 

User  enters  used 
new  Password/PIN 

The  user  enters  a  Password 
and/or  PIN  that  has  been 
used  in  any  of  the  10 
previous  password/PIN 
changes 

System  prompts  the  user  with  message  "New 
password/PIN  cannot  have  been  used  in  any  of 
the  previous  10  attempts" 

User  enters  incorrect 
format  for  new 
password 

The  user  enters  a  password 
that  contains  less  than  10 
characters  or  does  not 
contain  at  least  two  of: 
special  characters,  numbers, 
uppercase  letters,  lowercase 
letters 

System  prompts  the  user  with  message 
"Incorrect  password  format" 

User  enters  the  name 
or  part  of  it  in  the 
Search  by  Name  box 
and  clicks  on  the 
Search  button 

User  enters  the  soldier's 
name  or  part  of  it  in  the 
search  by  name  text  box  and 
clicks  on  the  Search  By 

Name  button 

System  filters  the  candidate  list  to  show  only 
those  patients  whose  names  either  partly  or 
completely  match  the  search  string 

User  enters  the 
required  SSN  using 
the  number  pad  and 
clicks  the  Search 
button 

User  selects  the  candidate's 
SSN  using  the  number  pad 
that  pops-up  when  the  user 
clicks  on  the  Search  by  SSN 
field.  He/She  then  clicks  on 
the  Search  by  SSN  button. 

System  filters  the  candidate  list  to  show  only 
the  patient  whose  SSN  matches  the  entered 
string. 

User  selects  the 
required  candidate 
from  the  candidate 
list  and  clicks  on 

LOAD  FORM 

User  selects  the  required 
candidate's  name  from  the 
Candidate  list  and  clicks  on 
the  VIEW  EXAM  button 

System  displays  a  confirmation  page  with  the 
candidate's  name  in  big,  bold  letters  and  a 

YES  and  NO  button. 

User  clicks  YES  on 
the  confirmation 
page 

User  confirms  that  the  name 
of  the  candidate  and  if  it 
matches,  clicks  on  the  YES 
button 

System  loads  the  exams  list  for  the  candidate 
with  the  passed  &  failed  exams  marked 
appropriately 

User  clicks  NO  on 
the  confirmation 
page 

User  confirms  that  the  name 
of  the  candidate  and  if  it 
does  not  match,  clicks  on 
the  NO  button 

System  loads  the  candidates  list  for  the  current 
set  of  exam  dates  again 

List  of  all  13  exams 
is  displayed 

Once  the  user  verifies  the 
name  of  the  candidate,  a  list 
of  13  exams  is  displayed 

The  list  shows  all  13  exams  with  updated 
statuses  for  each  exam.  The  passed  exams 
are  grayed  out  and  disabled  with  a  tick  mark 
before  the  exam  name.  The  failed  exams  are 
displayed  with  a  red  cross  mark  before  the 
name  of  the  exam. 

Selection  of  an  exam 

user  select  an  exam  for  the 
candidate  and  presses  the 
exam  button/link 

An  exam  is  selected  and  the  form  for  the  exam 
is  displayed 

selection  of  a  failed 
exam  by  the  same 
examiner 

user  selects  a  failed  exam 
for  a  candidate  who  was 
earlier  failed  in  the  same 
exam  by  the  same  examiner 

System  displays  a  message  "Please  ask  the 
candidate  to  go  to  a  different  examiner  for  this 
exam" 

selection  of  a  failed 
exam  by  a  different 

user  selects  a  failed  exam 
for  a  candidate  who  was 

System  displays  a  message  "Candidate  has 
failed  this  exam  <X>  times".  When  the  user 
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examiner 

earlier  failed  in  the  same 
exam  by  a  different 
examiner 

clicks  OK,  it  opens  the  exam  form. 

selection  of  an  un¬ 
counseled  failed 
exam  by  a  different 
examiner 

user  selects  a  failed  exam 
for  a  candidate  who  was 
earlier  failed  in  the  same 
exam  by  a  different 
examiner  and  the  candidate 
has  not  yet  been  counseled 
by  the  coordinator  for  the 
failed  exam 

System  displays  a  message  "Candidate  has 
not  yet  been  counseled  for  the  exam.  Please 
ask  the  candidate  to  go  to  the  coordinator  to 
complete  counseling".  The  user  clicks  on  OK 
and  the  System  loads  the  list  of  candidates 
again. 

exam  form  is  loaded 

The  chosen  exam  form  is 
loaded 

System  loads  the  corresponding  exam  form. 

The  name  of  the  exam  is  displayed  on  top  of 
the  form.  The  name  of  the  candidate,  the  name 
of  the  evaluator  and  the  exam  date  are 
displayed  too.  The  timer  is  displayed  with  the 
start  time  button  alone  enabled.  Also  based  on 
the  attempt  number,  a  label  saying  RETEST 
for  the  2nd  attempt  and  RETEACH  for  the  3rd 
and  final  attempt  is  also  displayed  on  top.  The 
total  score  is  displayed  at  the  bottom  of  the 
questions  list.  The  instructions  buttons,  the 
scenario  buttons  (where  applicable)  and  the 
comments  buttons  are  also  enabled.  There 
should  be  a  signature  button  at  the  bottom(on 
the  2nd  page  on  two  page  exams) 

The  user  chooses  to 
view  the  instructions 
for  examiner 

The  user  clicks  on  the 
"Instructions  for  examiner" 
button 

The  system  displays  the  instructions  for  the 
examiner  in  a  full  screen  window  with  a  close 
button  on  top.  This  window  cannot  be  resized. 

The  user  chooses  to 
view  the  instructions 
for  candidate 

The  user  clicks  on  the 
"Instructions  for  candidate 
button 

The  system  displays  the  instructions  for  the 
candidate  in  a  full  screen  window  with  a  close 
button  on  top.  This  window  cannot  be  resized. 

the  user  chooses  to 
view  a  scenario  for 
the  exam 

The  user  chooses  a  scenario 
and  clicks  on  the  button  with 
the  corresponding  number. 

The  scenario  is  loaded  in  a  pop-up  window. 

The  window  always  stays  on  top  and  it  is 
possible  to  move  it  around  and  resize  it.  The 
corresponding  scenario  button  remains 
highlighted  on  either  page.  The  scenario 
window  is  hidden  whenever  a  message  is 
popped-up  for  the  user,  until  the  user  responds 
to  the  message. 

The  user  chooses  to 
cancel  out  of  the 
form 

The  user  chooses  to  cancel 
the  form  if  he  has  selected 
the  wrong  form  by  pressing 
the  cancel;  button  on  the 
top. 

The  system  prompts  the  user  to  confirm  "Do 
you  want  to  exit  the  exam?”  When  the  user 
clicks  "yes"  the  form  is  closed  and  the  user  is 
returned  to  the  exams  list  for  the  candidate. 

user  is  able  to  start 
the  timer 

The  user  choose  to  start  the 
timer  manually  by  pressing 
the  "start  time"  button 

The  timer  starts  to  track  the  time  taken  to 
complete  the  examination.  The  start  time 
button  is  disabled.  The  pause  time  and  stop 
time  buttons  are  enabled. 

user  is  able  to  start 
the  timer  on  either 
page 

The  user  choose  to  start  the 
timer  manually  by  pressing 
the  "start  time"  button  on 
either  page 

The  timer  starts  to  track  the  time  taken  to 
complete  the  examination.  The  start  time 
button  is  disabled.  The  pause  time  and  stop 
time  buttons  are  enabled.  This  is  reflected  on 
the  other  page  too. 

user  is  able  to  pause 

The  user  clicks  on  the 

The  timer  stops.  The  PAUSE  TIME  button  is 
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the  timer 

PAUSE  TIME  button  and  the 
timer  stops  ticking 

hidden  and  the  RESUME  TIME  is  visible.  The 
STOP  TIME  is  still  enabled. 

user  is  able  to  pause 
the  timer  on  either 
page 

The  user  clicks  on  the 

PAUSE  TIME  button  on 
either  page  and  the  timer 
stops  ticking 

The  timer  stops.  The  PAUSE  TIME  button  is 
hidden  and  the  RESUME  TIME  is  visible.  The 
STOP  TIME  is  still  enabled.  This  is  reflected  on 
the  other  page  too. 

user  is  able  to 
resume  the  timer 

The  user  clicks  on  the 
RESUME  TIME  button  and 
the  timer  stops  ticking 

The  timer  starts  again  where  it  was  paused. 

The  RESUME  TIME  button  is  hidden  and  the 
PAUSETIME  is  visible.  The  STOP  TIME  is  still 
enabled. 

user  is  able  to 
resume  the  timer  on 
either  page 

The  user  clicks  on  the 
RESUME  TIME  button  on 
either  page  and  the  timer 
stops  ticking 

The  timer  starts  again  where  it  was  paused. 

The  RESUME  TIME  button  is  hidden  and  the 
PAUSETIME  is  visible.  The  STOP  TIME  is  still 
enabled.  This  is  reflected  on  the  other  page 
too. 

user  is  able  to  stop 
the  timer  manually 

The  user  presses  the  "Stop 
Timer"  button  when  exam  is 
done. 

The  timer  stops.  All  timer  controls  are  disabled 

user  is  able  to  stop 
the  timer  manually 
on  either  page 

The  user  presses  the  "Stop 
Timer"  button  on  either  page 
when  exam  is  done. 

The  timer  stops.  All  timer  controls  are  disabled. 
This  is  reflected  on  the  other  page  too. 

user  grades  the 
candidate  for  an 
exam 

The  user  starts  grading  the 
candidate  for  the  selected 
exam. 

The  user  is  able  to  select  the  radio  buttons  to 
mark  the  candidate’s  response  to  a  question. 

Automatic 
calculation  of 
candidates  score  on 
an  exam 

As  user  keeps  on  marking 
the  Y/N  answers.  The 
system  automatically  adds  a 

1  mark  for  each  question 
answered  as  Y.  The  system 
subtracts  1  mark  for  each 
question  answered  as  N. 

The  marks  for  the  candidate  are  automatically 
calculated  and  displayed.  Refer  "Quantitative 
Statistics"  tab  to  see  the  minimum  passing 
score  required  to  PASS  the  examination. 

user  selects  the 
check  box  for  a 
"Critical  criteria" 

user  select  the  check  box  for 
a  critical  criteria  question 

The  critical  criteria  check  box  gets  checked 

For  the  forms  which 
span  over  multiple 
pages,  a  "Next" 
button  is  available  on 
the  first  page 

If  exam  got  multiple  pages, 
the  evaluator  clicks  the 
"next"  button  to  move  to  the 
next  page  of  the  form 

The  next  page  of  the  form  is  displayed  for  the 
user  to  grade  the  candidate  on.  The  previously 
graded  questions  are  correctly  populated. 

For  the  forms  which 
span  over  multiple 
pages,  a  "Back" 
button  is  available  on 
the  2nd  page 

If  exam  got  multiple  pages, 
the  evaluator  clicks  the 
"back"  button  to  move  to  the 
previous  page  of  the  form 

The  previous  page  of  the  form  is  displayed  for 
the  user  to  grade  the  candidate  on.  The 
previously  graded  questions  are  correctly 
populated. 

user  is  allowed  to 
enter/see  his 
comments 

user  clicks  on  VIEW/ADD 
Comments  and  a  pop  up  text 
box  opens 

The  user  is  able  to  enter  in  his  remarks  by 
either  appending  to  the  existing  comments  or 
replacing  the  existing  comments  or  by  adding 
new  comments.  He/she  can  also  view 
comments  he/she  entered  previously  for  the 
same  exam. 

user  is  able  to  enter 
long  comments 

user  enters  really  long 
comments  and  these  are 
accepted 

System  allows  users  to  enter  comments  up  to 

1 000  characters 

user  tries  to  submit 
an  incomplete  form 

user  clicks  on  submit  button 
to  submit  an  incomplete  form 

The  system  prompts  the  user  to  complete  the 
form  before  submitting  it.  The  non-completed 
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question  shall  displayed  in  RED 

user  tries  to  submit 
an  form  without 
signing 

user  clicks  on  submit  button 
without  signing  the  form 

The  system  prompts  the  user  for  signing  the 
form  before  submitting  it. 

User  enters  the 
wrong  PIN  while 
signing  the  form 

The  user  signs  the  form  and 
clicks  submit. 

The  system  prompts  the  user  to  enter  the 
correct  PIN 

User  enters  the 
correct  to  sign  the 
form 

The  user  signs  the  form  and 
clicks  submit. 

The  following  message  is  displayed,  "Digitally 
Signed  By: "  followed  by  the  name  of  the  user 
followed  by  the  time  and  date  of  signing,  in  the 
Signature  Box. 

user  completes  the 
grading  of  the  exam 
by  signing  the  form 
but  needs  to  revise 
the  grading 

The  user  signs  the  form  but 
decides  to  re-grade  the  form. 
So  he  clicks  on  any  of  the 
questions  to  re-grade 

The  system  clears  the  Digital  Signature  and 
form  is  available  for  re-grading 

user  completes  the 
grading  of  the  exam 
by  signing  the  form 
and  tries  to  submit 
after  digitally  signing 
the  form  with  his/her 
PIN 

The  user  signs  the  form  by 
entering  his/her  PIN  and 
clicks  submit. 

The  form  is  submitted  and  the  following 
message  is  displayed  -  "Exam  Details 
successfully  submitted".  Once  the  user  clicks 
on  the  OK  button,  the  form  is  closed  and  the 
user  is  returned  to  the  candidate  list.  Any  open 
scenario  pop-ups  are  also  closed.  The  user  is 
returned  to  the  candidate  list. 

user  completes  the 
grading  of  the  exam 
by  signing  the  form 
and  tries  to  submit 
after  digitally  signing 
the  form  with  his/her 
PIN 

The  user  signs  the  form  by 
entering  his/her  PIN  and 
clicks  submit. 

If  the  wireless  connection  is  lost,  the  same  is 
displayed  as  a  message  and  the  form  is  not 
submitted  or  closed.  Scenario  windows  are  not 
closed. 

User  clicks  on  sign 
or  submit 

User  clicks  on  the  sign  or 
submit  form  buttons. 

System  prompts  the  user  with  the  message  - 
"Timer  will  automatically  be  stopped  now". 

Once  the  user  clicks  OK,  the  timer  is  stopped 
and  all  timer  controls  are  disabled. 

All  candidates  who 
have  either  passed 
or  failed  the  entire 

ALL  SKILLS 
EXAMINATION  will 
be  removed  from  the 
candidate  list 

Any  candidate  who  has 
passed  5  mandatory  and  1 
random  exams  or  failed  the 
same  exam  thrice  or  failed  4 
different  exams  is  removed 
from  the  candidate  list.  The 
candidate  list  only  contains 
Pending  candidates 

System  displays  correct  data  on  the  list. 

User  enters  the 
wrong  PIN  while 
signing  the  form 

The  user  signs  the  form  and 
clicks  submit. 

The  system  prompts  the  user  to  enter  the 
correct  PIN 

User  enters  the 
correct  to  sign  the 
form 

The  user  signs  the  form  and 
clicks  submit. 

The  following  message  is  displayed,  "Digitally 
Signed  By: "  followed  by  the  name  of  the  user 
followed  by  the  time  and  date  of  signing,  in  the 
Signature  Box. 

user  completes  the 
grading  of  the  exam 
by  signing  the  form 
but  needs  to  revise 
the  grading 

The  user  signs  the  form  but 
decides  to  re-grade  the  form. 
So  he  clicks  on  any  of  the 
questions  to  re-grade 

The  system  clears  the  Digital  Signature  and 
form  is  available  for  re-grading 

user  completes  the 

The  user  signs  the  form  by 

The  form  is  submitted  and  the  following 
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grading  of  the  exam 
by  signing  the  form 
and  tries  to  submit 
after  digitally  signing 
the  form  with  his/her 
PIN 

entering  his/her  PIN  and 
clicks  submit. 

message  is  displayed  -  "Exam  Details 
successfully  submitted".  Once  the  user  clicks 
on  the  OK  button,  the  form  is  closed  and  the 
user  is  returned  to  the  candidate  list.  Any  open 
scenario  pop-ups  are  also  closed.  The  user  is 
returned  to  the  candidate  list. 

user  completes  the 
grading  of  the  exam 
by  signing  the  form 
and  tries  to  submit 
after  digitally  signing 
the  form  with  his/her 
PIN 

The  user  signs  the  form  by 
entering  his/her  PIN  and 
clicks  submit. 

If  the  wireless  connection  is  lost,  the  same  is 
displayed  as  a  message  and  the  form  is  not 
submitted  or  closed.  Scenario  windows  are  not 
closed. 

User  clicks  on  sign 
or  submit 

User  clicks  on  the  sign  or 
submit  form  buttons. 

System  prompts  the  user  with  the  message  - 
"Timer  will  automatically  be  stopped  now". 

Once  the  user  clicks  OK,  the  timer  is  stopped 
and  all  timer  controls  are  disabled. 

All  candidates  who 
have  either  passed 
or  failed  the  entire 

ALL  SKILLS 
EXAMINATION  will 
be  removed  from  the 
candidate  list 

Any  candidate  who  has 
passed  5  mandatory  and  1 
random  exams  or  failed  the 
same  exam  thrice  or  failed  4 
different  exams  is  removed 
from  the  candidate  list.  The 
candidate  list  only  contains 
Pending  candidates 

System  displays  correct  data  on  the  list. 

8.1.2  Examiner  Portal  Functionality 

The  table  below  shows  the  eventual  requirements  created,  the  testing  action  and  the  expected 
system  responses. 


Requirements 

Description 

Action 

Expected  Results 

Examiner  Login 

Enter  the  User  Id  and  password  and  hit 
submit  tab 

Gives  Examiner  access  to  the 
portal  and  brings  to  the  "Home" 
screen  of  the  portal 

View  Records  by 

Date  Range 

In  From  and  To  tabs  select  the  dates  as 
desired  to  view  the  records  and  hit  "Submit" 

Lists  all  the  exams  reviewed  in 
the  specified  time  line 

Export  the  records  to 
Excel 

Select  the  "Export  to  Excel"  link  in  the  top 
right  corner 

Open  the  records  in  an  excel  file 

View  and  Print  exam 
given  by  the  student 

Select  the  exam  intended  to  view  and  click 
on  “Print  Exam"  icon.  When  the  exam 
opens  select  File>Print 

Prints  the  selected  exam 

View  Records  by 
Month  and  Year 

In  Month  tab  select  the  month  desired  for 
viewing.  In  Year  tab  select  the  year 

Lists  all  the  exams  reviewed  in 
the  specified  month  and  year 

Export  the  records  to 
Excel 

Select  the  "Export  to  Excel"  link  in  the  top 
right  corner 

Open  the  records  in  an  excel  file 

View  and  Print  exam 
given  by  the  student 

Select  the  exam  intended  to  view  and  click 
on  “Print  Exam"  icon.  When  the  exam 
opens  select  File>Print 

Prints  the  selected  exam 

View  Records  by 

Date 

Select  the  specific  date  in  the  "Date"  tab  as 
required 

Lists  all  the  exams  reviewed  by 
the  examiner  on  that  specific 
date 
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Export  the  records  to 
Excel 

Select  the  "Export  to  Excel"  link  in  the  top 
right  corner 

Open  the  records  in  an  excel  file 

View  and  Print  exam 
given  by  the  student 

Select  the  exam  intended  to  view  and  click 
on  “Print  Exam"  icon.  When  the  exam 
opens  select  File>Print 

Prints  the  selected  exam 

Search  Record  by 

SSN 

Enter  the  SSN  #  searching  for  in  the  "SSN” 
tab  and  Last  Name  of  the  student  in  the 
"Last  Name"  tab 

Lists  the  records 

Export  the  records  to 
Excel 

Select  the  "Export  to  Excel"  link  in  the  top 
right  corner 

Open  the  records  in  an  excel  file 

View  and  Print  exam 
given  by  the  student 

Select  the  exam  intended  to  view  and  click 
on  “Print  Exam"  icon.  When  the  exam 
opens  select  File>Print 

Prints  the  selected  exam 

Sort  Records  by 
exams  passed 

The  green  "Passed"  radio  button  is  selected 

Lists  all  the  exams  that  were 
passed  by  the  students 
administered  by  the  instructor 

Sort  Records  by 
exams  failed 

The  red  "Failed"  radio  button  is  selected 

Lists  all  the  exams  that  were 
failed  by  the  students 
administered  by  the  instructor 

Sort  Records  by 

Exam  Name 

In  the  drop  down  menu  select  the  exam 
name  for  which  want  to  see  the  desired 
results 

Lists  all  the  exams  given  by  the 
instructor  with  the  selected  exam 
name 

Export  records  into 
Excel 

Select  the  "Export  to  Excel"  link  in  the  top 
right  corner  and  select  "Open" 

Open  the  records  in  an  excel 

Exam  Centers  Report 
based  on  date  range 

Select  Reports  from  left  menu  and  then 
select  Exam  Centers  Report  on  the  right.  In 
From  and  To  tabs  select  the  dates  as 
desired  to  view  the  reports  for.  Then  hit 
Submit  and  select  the  exam  for  which  need 
to  view  the  results 

Displays  the  3D  graph  with  the 
exam  center  report  value 

Export  the  Report  to 
Excel 

Select  Export  to  Excel  link  on  top  right 

Opens  the  report  in  Excel  file 

Export  the  Report  to 
PDF 

Select  Export  to  PDF  link  on  top  right 

Opens  the  report  in  .pdf  file 

The  coordinator  adds 
a  new  examiner 

The  coordinator  selects  Manage  Examiner 
icon  and  selects  Add  Examiner  link  and  fills 
in  the  credentials  Username,  Password, 

First  Name,  Middle  Name,  and  Last  Name 

New  examiner  is  added  to  the 
portal 

The  coordinator 
deletes  examiner 

The  coordinator  selects  the  examiner  he 
wishes  to  delete  and  clicks  on  the  red 

Delete  User  Account  icon 

The  examiner  account  is  deleted 
from  the  portal 

The  coordinator  adds 
a  commander 

The  coordinator  selects  Manage 

Commander  and  selects  Add  Commander 
and  fills  in  the  credentials  Username, 
Password,  First  Name,  Middle  Name,  and 
Last  Name 

New  commander  is  added  to  the 
portal 

The  coordinator 
deletes  commander 
user  account 

The  coordinator  selects  the  commander 
from  the  displayed  list  he  needs  to  delete 
and  clicks  on  the  red  Delete  User  Account 
icon.  The  system  prompts  for  logging 
password.  Enter  admin  password 

The  commander  account  is 
deleted 

The  coordinator  adds 
a  site  coordinator 

The  coordinator  selects  Manage  Site 
Coordinators  and  selects  Add  Site 
Coordinator  and  fills  in  Username, 

Password,  First  Name,  Middle  Name,  and 

New  site  coordinator  is  added 
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Last  Name 

The  coordinator 
deletes  the  site 
coordinator  user 
account 

The  coordinator  selects  the  site  coordinator 
from  the  displayed  list  he  needs  to  delete 
and  clicks  on  the  red  Delete  User  Account 
icon 

The  site  coordinator  user  profile 
is  deleted  from  the  portal 

The  coordinator  adds 
a  admin  to  the  portal 

The  coordinator  selects  Manage 
Administrators  and  selects  Add  Admin  and 
fills  in  Username,  Password,  First  Name, 
Middle  Name,  and  Last  Name 

New  administrator  is  added  to 
the  portal 

The  coordinator 
deletes  the 
administrator  deletes 
user  account  from 
the  portal 

The  coordinator  selects  the  administrator 
from  the  displayed  list  he  needs  to  delete 
and  clicks  on  the  red  Delete  User  Account 
icon 

The  administrator  user  profile  is 
deleted  from  the  portal 

Change  Account 
Password 

Select  Updater  Account  from  the  left  menu. 
Type  the  new  password  in  Password  tab 
and  retype  the  password  in  Confirm 

Password  tab 

Changes  the  user  password 
associated  with  the  account 

Logging  out  of  the 
Portal 

Select  Logout  from  the  left  menu 

Logs  out  of  the  portal 

8.1.3  Server  Console  Functionality 

The  table  below  shows  the  eventual  requirements  created,  the  testing  action  and  the  expected 
system  responses. 


Requirements 

Description 

Action 

Expected  Results 

The  site  administrator 
logs  into  the  console 
with  invalid 
credentials 

The  administrator  enter  incorrect  User  Id 
and  password 

The  system  prompts  the  user  with 
message  "Account  inactive.  Please 
contact  administrator" 

The  site  administrator 
logs  into  the  console 
with  correct 
credentials 

The  administrator  enter  correct  User  Id 
and  password 

The  administrator  is  logged  in 

Administrator  logs 
out 

The  administrator  selects  "Manage 
users->logout" 

The  administrator  is  logged  out 

Administrator  exits 
the  application 

The  administrator  selects  "Manage 
users->exit" 

The  system  logs  the  administrator 
out  and  exits  the  application. 

The  administrator 
imports  the 
scheduled  users  for 
the  day 

The  administrator  imports  the  user  by 
selecting  the  "Manage  users->manager 
candidates->import  candidates"  option 
and  selecting  the  import  excel  file 

The  users  are  successfully  imported 
into  the  system  and  displayed  in  the 
view  candidate  section 

The  administrator 
adds  a  new  candidate 
to  schedule  for  the 
day 

The  administrator  clicks  on  add  user  and 
enters  the  details(SSN,  First  Name, 

Middle  name,  Last  name,  group  name, 
team  name,  selects  the  schedule  date" 
and  clicks  add 

The  new  candidate  is  added  to  the 
system  for  the  current  day. 

The  administrator 
chooses  to  modify  the 
profile  of  an  existing 
candidate 

The  administrator  selects  a  user  from  the 
view  candidate  screen  and  clicks  edit. 

He  then  updates  the  necessary  fields 
and  clicks  update 

The  user  details  are  modified  and 
persisted  in  the  system 

The  administrator 
adds  an 

The  administrator  selects  the  option  for 
add  examiner.  He  fills  in  the  necessary 

The  evaluator  is  added  to  the 
system  and  the  list  of  evaluators  is 
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evaluator/examiner 

details(SSN,  First  Name,  Middle  name, 
Last  name,  user  name,  password)  and 
clicks  add 

displayed  on  the  manage 
evaluators  screen 

The  administrator 
while  adding  the 
evaluator  enters  a 
user  name  which 
already  exists 

The  administrator  adds  an  evaluator  but 
enters  a  user  name  which  already  exists 
in  the  system 

The  system  prompts  the  user  with 
message  "username  already  exists, 
please  choose  another  user  name" 

The  administrator 
edits  an  evaluators 
profile 

The  administrator  selects  an  evaluator 
under  View  Examiner  option  and  clicks 
edit.  He  changes  the  necessary  details 
and  clicks  update 

The  details  of  the  evaluator  are 
modified 

The  administrator 
deletes  an  evaluators 
profile 

The  administrator  selects  an  evaluator 
under  View  Examiner  and  clicks  delete 

The  evaluator  is  deleted  from  the 
system 

The  administrator 
adds  a  coordinator 

The  administrator  selects  the  option  for 
add  coordinator.  He  fills  in  the  necessary 
details(SSN,  First  Name,  Middle  name, 
Last  name,  user  name,  password)  and 
clicks  add 

The  coordinator  is  added  to  the 
system  and  the  list  of  evaluators  is 
displayed  on  the  view  coordinator 
screen 

The  administrator 
while  adding  the 
coordinator  enters  a 
user  name  which 
already  exists 

The  administrator  adds  a  coordinator  but 
enters  a  user  name  which  already  exists 
in  the  system 

The  system  prompts  the  user  with 
message  "username  already  exists, 
please  choose  another  user  name" 

The  administrator 
edits  a  coordinator's 
profile 

The  administrator  selects  a  coordinator 
and  clicks  edit.  He  changes  the 
necessary  details  and  clicks  update 

The  details  of  the  coordinator  are 
modified 

The  administrator 
deletes  a  coordinator 
profile 

The  administrator  selects  a  coordinator 
and  clicks  delete 

The  coordinator  is  deleted  from  the 
system 

The  administrator 
refreshes  the  manage 
coordinator  screen 

The  administrator  on  the  manage 
coordinator  screen  clicks  the  refresh 
button 

The  manage  coordinator  screen  is 
refreshed  from  the  database  with 
latest  information 

The  coordinator 
chooses  to 
synchronize  exams 
but  provides  incorrect 
credentials  or 
credentials  which 
don't  exist  on  the 
portal 

The  coordinator  selects  synchronize 
exams  and  can  choose  to  synchronize 
either  each  exam  individually  or  all  at  the 
same  time  by  selecting  Sync  All  but 
provides  incorrect  credentials  or 
credentials  which  don't  exist  on  the  portal 

The  system  responds  with  error 
message  communicating  to  the 
user  that  the  account  doesn't  exist 
and  the  credentials  are  invalid 

The  coordinator 
chooses  to 
synchronize  exams 
and  provides  correct 
credentials 

The  coordinator  selects  synchronize 
exams  and  can  choose  to  synchronize 
either  each  exam  individually  or  all  at  the 
same  time  by  selecting  Sync  All  and 
provides  correct  credentials 

The  exams  are  synchronized  with 
the  EMT  server 

The  coordinator 
chooses  to  view  all 
previously 
synchronized  exams 

The  coordinator  in  Synchronize  Exams 
tab  selects  Synchronized  Exams 

The  system  displays  list  of  all 
previously  synchronized  exams 
along  with  users  name,  score,  and 

P/F  info 

The  coordinator 
chooses  to  delete  1 
synchronized  exam 

The  coordinator  selects  1  exam  from  the 
list  of  synchronized  exams  and  clicks 
Delete  button 

The  selected  synchronized  exam 
gets  deleted 

The  coordinator 

The  coordinator  clicks  Delete  Sync 

The  system  prompts  the  user  to 
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chooses  to  delete  all 
synchronized  exam 

button 

confirm  that  he  wants  to  delete  all 
synchronized  exams.  Once  you 
user  confirms  by  clicking  yes,  all  the 
synchronized  exam  get  deleted 

Maximum 

Synchronization  Time 

The  synchronization  should  complete  in 
Maximum  60  minutes 

The  synchronization  completes  in 
Maximum  60  minutes 

The  administrator 
chooses  to  view  the 
passed  candidates 
and  searches  by  last 
name 

The  administrator  clicks  on  "exams  and 
candidates->passed  candidates”.  He 
enters  last  name  of  a  candidate  and 
clicks  the  find  button 

The  system  displays  the  details  of 
the  candidate 

The  administrator 
chooses  to  view  the 
passed  candidates 
and  searches  by  a 
particular  date 

The  administrator  clicks  on  "exams  and 
candidates->passed  candidates”.  He 
enters/selects  a  date  and  clicks  the  find 
button 

The  system  displays  the  list  of  the 
candidates 

The  administrator 
chooses  to  view  the 
passed  candidates 
and  searches  by  a 
range  of  date 

The  administrator  clicks  on  "exams  and 
candidates->passed  candidates”.  He 
selects  a  range  date  by  providing  the 
values  in  "From"  and  "To"  and  clicks  the 
find  button 

The  system  displays  the  list  of  the 
candidates 

The  administrator 
chooses  to  view  the 
list  of  passed  exams 
of  a  candidate 

The  administrator  selects  a  candidate 
and  clicks  "View  Exam" 

The  system  displays  the  list  of 
passed  exams  for  a  candidate 

The  administrator 
chooses  to  view  the 
failed  candidates  and 
searches  by  last 
name 

The  administrator  clicks  on  "exams  and 
candidates->failed  candidates" .  He 
enters  last  name  of  a  candidate  and 
clicks  the  find  button 

The  system  displays  the  details  of 
the  candidate 

The  administrator 
chooses  to  view  the 
failed  candidates  and 
searches  by  a 
particular  date 

The  administrator  clicks  on  "exams  and 
candidates->failed  candidates" .  He 
enters/selects  a  date  and  clicks  the  find 
button 

The  system  displays  the  list  of  the 
candidates 

The  administrator 
chooses  to  view  the 
failed  candidates  and 
searches  by  a  range 
of  date 

The  administrator  clicks  on  "exams  and 
candidates->failed  candidates”.  He 
selects  a  range  date  by  providing  the 
values  in  "From"  and  "To"  and  clicks  the 
find  button 

The  system  displays  the  list  of  the 
candidates 

The  administrator 
chooses  to  view  the 
list  of  failed  exams  of 
a  candidate 

The  administrator  selects  a  candidate 
and  clicks  "View  Exam" 

The  system  displays  the  list  of  failed 
exams  for  a  candidate 

The  administrator 
chooses  to  view  the 
pending  candidates 
and  searches  by  last 
name 

The  administrator  clicks  on  "exams  and 
candidates->pending  candidates”.  He 
enters  last  name  of  a  candidate  and 
clicks  the  find  button 

The  system  displays  the  details  of 
the  candidate 

The  administrator 
chooses  to  view  the 
pending  candidates 
and  searches  by  a 
particular  date 

The  administrator  clicks  on  "exams  and 
candidates->pending  candidates”.  He 
enters/selects  a  date  and  clicks  the  find 
button 

The  system  displays  the  list  of  the 
candidates 

The  administrator 

The  administrator  clicks  on  "exams  and 

The  system  displays  the  list  of  the 
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chooses  to  view  the 
pending  candidates 
and  searches  by  a 
range  of  date 

candidates->pending  candidates”.  He 
selects  a  range  date  by  providing  the 
values  in  "From"  and  "To”  and  clicks  the 
find  button 

candidates 

The  administrator 
chooses  to  view  the 
list  of  pending  exams 
of  a  candidate 

The  administrator  selects  a  candidate 
and  clicks  "View  Exam” 

The  system  displays  the  list  of 
pending  exams  for  a  candidate 

The  administrator 
chooses  to  view  the 
current  EMT 
information 

The  administrator  chooses  "Help->About 
EMT" 

The  systems  displays  the  details 
about  the  current  EMT  application 
like  "version  etc" 

The  coordinator 
chooses  to  delete 
previous 
synchronization 

The  coordinator  selects  the  exam  from 
the  list  of  listed  previously  synchronized 
exams  that  he  wishes  to  delete  and 
selects  Delete  Synchronization 

The  system  deletes  the  selected 
exam  from  the  displayed  list  of 
synchronized  exams 

The  coordinator 
chooses  to  delete 
exam 

The  coordinator  selects  the  exam  that  he 
wishes  to  delete  and  selects  Delete 

Exam 

The  system  deletes  the  selected 
exam  from  the  list  of  exams. 

The  coordinator  views 
report  of  all  passed 
exams 

Selects  View  Exam  Reports>AII 
Exams>Passed  and  fills  in  the  search 
criteria  i.e.  date  needed  to  view  the 
reports  of  the  passed  exams 

Lists  all  the  exams  passed  in  the 
specified  timeline 

The  coordinator  filters 
the  passed  exam 
records  by 
examination  name 

Selects  the  exam  to  sort  by  from  the 
options  listed  in  Filter  Record  by 
Examination  Exam 

Lists  the  selected  exam  that  were 
passed  in  the  predefined  timeline 

The  coordinator 
chooses  to  view  the 
reports  of  passed 
exam  by  Group  Name 

Selects  the  group  name  the  coordinator 
intends  to  view  reports  for  from  the  listed 
group  names  in  Group  Name  drop  down. 

Lists  all  the  passed  exams  by 
candidates  pertaining  to  the  group 
selected  in  the  search  criteria 

The  coordinator 
chooses  to  view  the 
reports  of  passed 
exam  by  Team  Name 

Selects  the  team  name  the  coordinator 
intends  to  view  reports  for  from  the  listed 
team  names  in  Team  Name  drop  down. 

Lists  all  the  passed  exams  by 
candidates  pertaining  to  the  team 
selected  in  the  search  criteria 

The  coordinator 
exports  the  report  to 
excel 

Selects  the  export  to  excel  icon 

The  system  opens  MS  Excel 
spreadsheet  with  the  exported  data 

The  coordinator 
chooses  to  rescore  an 
exam 

Selects  the  exam  intended  to  rescore 
and  selects  view  exam  icon.  Changes 
the  points  awarded  as  required.  Enters 
the  reason  to  rescore  in  the  box  and 
selects  rescore  box 

The  system  rescores  the  exam 

The  coordinator  views 
the  reports  of  all 
failed  exams 

Selects  View  Exam  Reports>AII 
Exams>Failed  and  fills  in  the  search 
criteria  i.e.  date  needed  to  view  the 
reports  of  the  failed  exams 

Lists  all  the  exams  failed  in  the 
specified  timeline 

The  coordinator  filters 
the  passed  exam 
records  by 
examination  name 

Selects  the  exam  to  sort  by  from  the 
options  listed  in  Filter  Record  by 
Examination  Name 

Lists  the  selected  exam  that  were 
passed  in  the  predefined  timeline 

The  coordinator 
chooses  to  view  the 
reports  of  failed 
exams  by  Group 

Selects  the  group  name  the  coordinator 
intends  to  view  reports  for  from  the  listed 
group  names  in  Group  Name  drop  down. 

Lists  all  the  failed  exams  by 
candidates  pertaining  to  the  group 
selected  in  the  search  criteria 

Deloitte-Federal  Healthcare 


40  of  57 


Final  Report  /  Concept  of  Operations  ( CONOPS ) 


DCMT  TE  W81 XWH-07-C-01 1 5 


Name 

The  coordinator 
chooses  to  view  the 
reports  of  failed  exam 
by  Team  Name 

Selects  the  team  name  the  coordinator 
intends  to  view  reports  for  from  the  listed 
team  names  in  Team  Name  drop  down. 

Lists  all  the  failed  exams  by 
candidates  pertaining  to  the  team 
selected  in  the  search  criteria 

The  coordinator 
exports  the  report  to 
excel 

Selects  the  export  to  excel  icon 

The  system  opens  MS  Excel 
spreadsheet  with  the  exported  data 

The  coordinator 
wishes  to  view  the 
exam  taken  by 
candidates 

Selects  the  exam  wished  to  view  and 
selects  view  exam  icon 

The  system  displays  the  exam 
selected  for  viewing 

The  coordinator 
chooses  to  rescore  an 
exam 

Selects  the  exam  intended  to  rescore 
and  selects  view  exam  icon.  Changes 
the  points  awarded  as  required.  Enters 
the  reason  to  rescore  in  the  box  and 
selects  rescore  box 

The  system  rescores  the  exam 

The  coordinator 
chooses  to  view  the 
reports  of  all  exams 

Selects  View  Exam  Reports>AII 

Exams>AII  and  fills  in  the  search  criteria 
i.e.  date  needed  to  view  the  reports  of  all 
the  exams 

The  system  displays  all  the  exams- 
passed  and  failed  taken  in  the 
selected  timeline 

The  coordinator  filters 
the  exam  records  by 
examination  name 

Selects  the  exam  to  sort  by  from  the 
options  listed  in  Filter  Record  by 
Examination  Exam 

Lists  the  selected  exam  that  were 
passed  in  the  predefined  timeline 

The  coordinator 
chooses  to  view  the 
reports  of  all  exam  by 
Group  Name 

Selects  the  group  name  the  coordinator 
intends  to  view  reports  for  from  the  listed 
group  names  in  Group  Name  drop  down. 

Lists  all  the  exams  by  candidates 
pertaining  to  the  group  selected  in 
the  search  criteria 

The  coordinator 
chooses  to  view  the 
reports  of  all  exams 
by  Team  Name 

Selects  the  team  name  the  coordinator 
intends  to  view  reports  for  from  the  listed 
team  names  in  Team  Name  drop  down. 

Lists  all  exams  by  candidates 
pertaining  to  the  team  selected  in 
the  search  criteria 

The  coordinator 
exports  the  report  to 
excel 

Selects  the  export  to  excel  icon 

The  system  opens  MS  Excel 
spreadsheet  with  the  exported  data 

The  coordinator 
wishes  to  view  the 
exam  taken  by 
candidates 

Selects  the  exam  wished  to  view  and 
selects  view  exam  icon 

The  system  displays  the  exam 
selected  for  viewing 

The  coordinator 
chooses  to  view  the 
exam  statistics 

Specifies  the  dates  between  which  the 
coordinator  intends  to  view  statistics  for 
and  selects  pass  or  fail  as  desired 

Lists  the  statistics  of  all  the  exams 
taken  during  the  time  frame 

The  coordinator  views 
the  statistic  based  on 
Group  Name 

The  coordinator  selects  the  group  name 
to  view  the  statistics  of  from  Group 

Name  drop  down 

Lists  the  selected  group's 
performance  statistics 

The  coordinator  views 
the  statistic  based  on 
Team  Name 

The  coordinator  selects  the  team  name 
to  view  the  statistics  of  from  Team  Name 
drop  down 

Lists  the  selected  team's 
performance  statistics 

The  coordinator 
chooses  to  export  the 
report  to  excel 

Selects  the  export  to  excel  icon 

The  system  opens  MS  Excel 
spreadsheet  with  the  exported  data 

The  coordinator  views 
the  exam  statistics  for 
all  the  exams  taken 
on  a  particular  date 

In  By  Date  tab  selects  the  date  for  which 
he  wishes  to  the  results  for  in  Date 

Lists  all  the  exams  taken  on  the 
selected  exams 
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The  coordinator  views 
the  statistic  based  on 
Group  Name 

The  coordinator  selects  the  group  name 
to  view  the  statistics  of  from  Group 

Name  drop  down 

Lists  the  selected  group's 
performance  statistics  on  the 
specific  date 

The  coordinator  views 
the  statistic  based  on 
Team  Name 

The  coordinator  selects  the  team  name 
to  view  the  statistics  of  from  Team  Name 
drop  down 

Lists  the  selected  team's 
performance  statistics  on  the 
specific  date 

The  coordinator 
chooses  to  export  the 
report  to  excel 

Selects  the  export  to  excel  icon 

The  system  opens  MS  Excel 
spreadsheet  with  the  exported  data 

The  coordinator  views 
All  Exams  taken  till 
date 

Selects  All  Exams  link  under  Exam-wide 
Statistics 

Lists  all  the  exams  taken  till  date 

The  coordinator  views 
the  statistic  based  on 
Group  Name 

The  coordinator  selects  the  group  name 
to  view  the  statistics  of  from  Group 

Name  drop  down 

Lists  the  selected  group's 
performance  statistics  till  date 

The  coordinator  views 
the  statistic  based  on 
Team  Name 

The  coordinator  selects  the  team  name 
to  view  the  statistics  of  from  Team  Name 
drop  down 

Lists  the  selected  team's 
performance  statistics  till  date 

Coordinator  exports 
the  statistics  to  excel 

Selects  the  intended  statistics  and 
selects  Export  to  Excel 

The  system  opens  MS  Excel 
spreadsheet  with  the  exported  data 

Coordinator  views  the 
rescored  exams 
between  two  specified 
dates 

Defines  the  dates  from  and  to  for  which 
intended  to  view  the  result  for  under 
Between  Dates 

The  system  displays  all  the  exams 
rescored  in  the  specified  timeline 

Coordinator  chooses 
to  view  the  rescored 
exams  based  on 
Examination  name 

Selects  the  exam  to  sort  by  from  the 
options  listed  in  Filter  Record  by 
Examination  Name 

Lists  all  the  rescored  exams  with 
the  specified  Exam  name  between 
the  predefined  dates 

Coordinator  chooses 
to  view  the  rescored 
exams  based  on 

Group  Name 

Selects  the  group  name  the  coordinator 
intends  to  view  reports  for  from  the  listed 
group  names  in  Group  Name  drop  down. 

Lists  all  the  rescored  exams  with 
the  specified  Group  name  between 
the  predefined  dates 

Coordinator  chooses 
to  view  the  rescored 
exams  based  on 

Team  Name 

Selects  the  team  name  the  coordinator 
intends  to  view  reports  for  from  the  listed 
group  names  in  Team  Name  drop  down. 

Lists  all  the  rescored  exams  with 
the  specified  team  name  between 
the  predefined  dates 

The  coordinator 
exports  the  report  to 
excel 

Selects  the  export  to  excel  icon 

The  system  opens  MS  Excel 
spreadsheet  with  the  exported  data 

The  coordinator 
wishes  to  view  the 
exam  taken  by 
candidates 

Selects  the  exam  wished  to  view  and 
selects  view  exam  icon 

The  system  displays  the  exam 
selected  for  viewing 

Coordinator  views  the 
rescored  exams  on  a 
specific  date 

Selects  the  date  for  which  intended  to 
view  the  result  under  By  Date 

The  system  displays  all  the  exams 
rescored  on  the  specific  date 

Coordinator  chooses 
to  view  the  rescored 
exams  based  on 
Examination  name 

Selects  the  exam  to  sort  by  from  the 
options  listed  in  Filter  Record  by 
Examination  Name 

Lists  all  the  rescored  exams  with 
the  specified  Exam  name  on  the 
specific  date 

Coordinator  chooses 
to  view  the  rescored 
exams  based  on 

Group  Name 

Selects  the  group  name  the  coordinator 
intends  to  view  reports  for  from  the  listed 
group  names  in  Group  Name  drop  down. 

Lists  all  the  rescored  exams  with 
the  specified  Group  name  on  the 
specific  date 

Coordinator  chooses 

Selects  the  team  name  the  coordinator 

Lists  all  the  rescored  exams  with 
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to  view  the  rescored 
exams  based  on 

Team  Name 

intends  to  view  reports  for  from  the  listed 
group  names  in  Team  Name  drop  down. 

the  specified  team  name  on  the 
specific  date 

The  coordinator 
exports  the  report  to 
excel 

Selects  the  export  to  excel  icon 

The  system  opens  MS  Excel 
spreadsheet  with  the  exported  data 

The  coordinator 
wishes  to  view  the 
exam  taken  by 
candidates 

Selects  the  exam  wished  to  view  and 
selects  view  exam  icon 

The  system  displays  the  exam 
selected  for  viewing 

Coordinator  views  all 
rescored  exams  till 
date 

Selects  All  Rescored  under  Rescored 
exams 

The  system  displays  all  the  exams 
rescored  all  the  exams  till  date 

Coordinator  chooses 
to  view  the  rescored 
exams  based  on 
Examination  name 

Selects  the  exam  to  sort  by  from  the 
options  listed  in  Filter  Record  by 
Examination  Name 

Lists  all  the  rescored  exams  with 
the  specified  Exam  name  till  date 

Coordinator  chooses 
to  view  the  rescored 
exams  based  on 

Group  Name 

Selects  the  group  name  the  coordinator 
intends  to  view  reports  for  from  the  listed 
group  names  in  Group  Name  drop  down. 

Lists  all  the  rescored  exams  with 
the  specified  Group  name  till  date 

Coordinator  chooses 
to  view  the  rescored 
exams  based  on 

Team  Name 

Selects  the  team  name  the  coordinator 
intends  to  view  reports  for  from  the  listed 
group  names  in  Team  Name  drop  down. 

Lists  all  the  rescored  exams  with 
the  specified  team  name  till  date 

The  coordinator 
exports  the  report  to 
excel 

Selects  the  export  to  excel  icon 

The  system  opens  MS  Excel 
spreadsheet  with  the  exported  data 

The  coordinator 
wishes  to  view  the 
exam  taken  by 
candidates 

Selects  the  exam  wished  to  view  and 
selects  view  exam  icon 

The  system  displays  the  exam 
selected  for  viewing 

Coordinator  chooses 
to  view  the  rescore 
statistics  between  two 
dates 

Specifies  dates  in  From  and  To  under 
Between  dates  menu  in  the  Rescore 
Statistics 

The  system  displays  the  statistics  of 
rescored  exams  during  the 
specified  timeline 

The  coordinator  views 
the  rescore  statistic 
based  on  Group 

Name 

The  coordinator  selects  the  group  name 
to  view  the  statistics  of  from  Group 

Name  drop  down 

Lists  the  selected  group's  rescore 
statistics  between  the  two 
predefined  dates 

The  coordinator  views 
the  statistic  based  on 
Team  Name 

The  coordinator  selects  the  team  name 
to  view  the  statistics  of  from  Team  Name 
drop  down 

Lists  the  selected  team's  rescore 
statistics  b/w  the  two  predefined 
dates 

The  coordinator 
chooses  to  export  the 
report  to  excel 

Selects  the  export  to  excel  icon 

The  system  opens  MS  Excel 
spreadsheet  with  the  exported  data 

Coordinator  chooses 
to  view  the  rescore 
statistics  for  a 
specific  date 

Chooses  the  date  for  which  wished  to 
view  the  statistics  of  rescored  exams 
under  the  By  Date 

Lists  the  statistics  of  rescored 
exams  on  the  specified  date 

The  coordinator  views 
the  statistic  based  on 
Group  Name 

The  coordinator  selects  the  group  name 
to  view  the  statistics  of  from  Group 

Name  drop  down 

Lists  the  selected  group's  rescore 
statistics  on  the  specific  date 

The  coordinator  views 
the  statistic  based  on 

The  coordinator  selects  the  team  name 
to  view  the  statistics  of  from  Team  Name 

Lists  the  selected  team's  rescore 
statistics  on  the  specific  date 
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Team  Name 

drop  down 

The  coordinator 
chooses  to  export  the 
report  to  excel 

Selects  the  export  to  excel  icon 

The  system  opens  MS  Excel 
spreadsheet  with  the  exported  data 

Coordinator  views 
rescore  statistics  of 
all  exams  rescored  till 
date 

Selects  All  Exams  link  under  Rescore 
Statistics 

Lists  the  statistics  of  all  rescored 
exams  till  date 

The  coordinator  views 
the  statistic  based  on 
Group  Name 

The  coordinator  selects  the  group  name 
to  view  the  statistics  of  from  Group 

Name  drop  down 

Lists  the  selected  group's  rescore 
statistics  till  date 

The  coordinator  views 
the  statistic  based  on 
Team  Name 

The  coordinator  selects  the  team  name 
to  view  the  statistics  of  from  Team  Name 
drop  down 

Lists  the  selected  team's  rescore 
statistics  till  date 

Coordinator  exports 
the  statistics  to  excel 

Selects  the  intended  statistics  and 
selects  Export  to  Excel 

The  system  opens  MS  Excel 
spreadsheet  with  the  exported  data 

The  coordinator 
chooses  to  view 
evaluator  reports  as  a 
daily  report 

Defines  the  date  for  which  the 
coordinator  intends  to  view  the  report  for 
in  Date  under  Daily  Reports 

Lists  the  evaluator  reports  for  the 
defined  date 

The  coordinator 
chooses  to  view  the 
daily  report  sorted  by 
Evaluator's  name 

The  coordinator  selects  evaluator's  name 
to  view  the  statistics  for 

Lists  the  selected  evaluator's  daily 
reports 

The  coordinator 
chooses  to  export  the 
report  to  excel 

Selects  the  export  to  excel  icon 

The  system  opens  MS  Excel 
spreadsheet  with  the  exported  data 

8.1.4  Preliminary  Test  Results 

The  Deloitte  Team  conducted  system  tests  beginning  with  the  April  1,  results  below 


Section 

Number  of 
Test  Areas 

Passed 

Failed 

Not  Tested 

Evaluator/Table  PC 

28 

28 

0 

0 

Portal 

74 

45 

19 

10 

EMT  Console 

121 

104 

8 

9 

Total  Test  Cases 

223 

177 

27 

19 

Continual  testing  and  development  conducted  until  the  functionality  listed  above  were  complete. 
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8.2  December  2009  Field  Testing 

Deloitte  conducted  a  train  the  trainer  session  and  performed  field  system  testing  during  a  four 
day  period.  The  following  personnel  were  in  attendance. 

DCMT  Personnel: 

•  Mr.  Marcus  Vigil 

•  Ms.  Chavez 

•  CPT  Brookins 

•  DCMT  Instructors 

Deloitte  Personnel: 

•  Mr.  Pawan  Singh 

•  Mr.  Raman  Singh 

•  Mr.  Devchand  Chute 

•  Mr.  Shivaraman  Raghuraman 


8.2.1  Day  1  Environment  and  Setups 

Setup  64  Tablet  PCs  for  use  in  the  ALL  SKILLS  examination. 

We  set  up  the  Tablet  PC  application  on  64  Tablet  PCs.  We  added  the  mapping  between  the 
server  name  and  its  static  IP  address  into  all  the  Tablet  PC’s.  Each  Tablet  was  completely 
charged.  We  also  installed  and  configured  the  EMT  Console  application  on  the  local  server 
laptop  along  with  the  Application  and  Database  on  the  laptop.  The  laptop  was  configured  with  a 
static  IP  address. 

Setup  Wireless  Environment 

The  secure  wireless  environment  was  set  up  at  the  exam  rooms  in  the  following  manner.  We 
setup  the  Laptop  Server  with  the  Console  application  in  the  coordinators  room.  (  Conf  Room 
120).  A  wireless  controller  acting  as  the  main  wireless  gateway  was  also  setup  in  the  same 
room.  The  laptop  server  was  connected  to  the  gateway  using  an  Ethernet  cable.  In  each  of  the 
exam  rooms,  one  wireless  access  points  was  setup  to  enhance  the  wireless  connectivity  for  all  the 
Tablet  PCs. 


8.2.2  Day  2  Training 

The  scope  of  training  provided  included: 

•  Train  users  on  the  process  and  data  flow  of  the  solution  including  Tablet  PC,  Console  and 
Portal  application. 

•  Demonstrate  Tablet  PC  application  including  Exam  navigation,  completion,  signing  and 
submission. 
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•  Demonstrate  Console  application  including  Reporting,  Dashboard,  Re-Score  and 
synchronization  functionality. 

•  Demonstrate  Web  Portal  application  including  data  verification,  reporting  and  printing. 

Training  conducted: 

•  Around  40  Instructors/instructors  were  trained  in  a  quick  training  session  which  covered 
the  following: 

•  Demonstrated  Tablet  PC  application  including  form  navigation,  completion,  signing  and 
submission. 

•  Demonstrated  Console  application  including  Reporting  and  Dashboard  functionality. 

•  We  also  answered  questions  about  the  ability  to  write  on  the  tablet  PCs. 

•  The  instructors  were  then  given  a  Tablet  PC  each  to  practice. 

•  A  training  database  was  setup  with  dummy  candidates  and  instructors 

•  The  instructors  were  allowed  to  complete  forms  and  submit  exams  for  these  dummy 
candidates,  so  as  to  get  a  feel  of  the  exam. 

Issues  encountered  and  corrected 

During  the  training  exercise,  we  uncovered  an  issue.  When  more  than  6  users  submitted  forms  at 
the  same  exact  time,  a  deadlock  condition  arose.  To  fix  this  issue,  we  modified  the  Tablet  PC 
application  to  listen  to  the  deadlock  error.  Upon  the  occurrence  of  the  error,  we  resubmitted  the 
form  data  automatically  until  the  submission  was  successful.  This  fixed  the  issue. 

8.2.3  Day  3  Field  Testing 

Instructors  Trial  Testing 

The  instructors  were  assigned  one  Tablet  PC  each  and  we  signed  the  Tablet  PCs  out.  The 
candidates  were  listed  in  the  Tablet  PC  application  and  the  evaluators  were  able  to  pick  them  out 
easily.  The  instructors  used  the  various  search  features  to  select  the  candidate,  and  a  list  of 
exams  available  for  each  candidate  was  shown.  The  instructors  loaded  the  skill  sheet  for  the 
required  exam  and  graded  the  candidate  by  starting  the  timer.  The  instructors  were  also  able  to 
read  out  instructions  to  the  candidates  by  bringing  up  the  instructions  on  the  application.  The 
instructors  were  also  able  to  bring  up  scenarios,  when  necessary,  on  the  Tablet  PC  using  the 
application 

Coordinators  Trial  Testing 

The  coordinator  was  able  to  track  exams  as  they  were  being  submitted.  The  coordinator  was  able 
to  track  candidates  who  failed  exams  and  was  able  to  counsel  them  immediately  and  send  them 
to  a  Re-Teach  session,  if  necessary.  The  coordinator  was  also  able  to  track  if  any  candidate  failed 
the  ALL  SKILLS  EXAMINATION,  on  an  immediate  basis  and  he  coordinator  was  able  to 
generate  reports  as  necessary. 

Testing  Results 

A  total  of  2790  exams  were  submitted  by  the  end  of  the  All  Skills  Examination.  (This  includes 
the  passed  and  failed  exams).  The  coordinator  had  very  little  work  except  to  click  buttons  to 
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generate  the  required  reports  at  the  end  of  the  examination.  Instructors  found  the  Tablet  PC 
application  much  easier  and  faster  to  use  vs.  the  paper  based  forms.  Instructors  noticed  the  time 
saved  in  pulling  up  the  forms  on  Tablet  PC  with  all  the  candidate  information  pre-populated. 
Instructors  felt  comfortable  writing  comments  on  the  Tablet  PC.  Instructors  also  liked  the  auto- 
populated  comments  for  critical  criteria,  which  reduced  the  amount  of  redundant  typing  required 
on  their  part. 

Instructors  also  noticed  that  the  application  and  wireless  environment  were  very  stable  with 
absolutely  no  issues  on  the  second  day  of  All  Skills.  Battery  life  of  Tablet  PC  was  down  to 
anywhere  from  65%  to  80%  after  3  hours  of  usage.  Instructors  noticed  they  saved  time  in  not 
having  to  count  and  grade  the  exam  manually,  as  the  application  did  that  for  them  automatically. 

Some  issues  pointed  out  to  us  by  the  instructors  and  the  coordinator  were: 

•  Need  for  count  of  exams  by  attempt  for  the  candidates. 

•  Need  to  fix  some  typos  in  the  instructions  and  the  scenarios. 

•  Overall  the  feedback  was  very  positive. 


8.2.4  Additional  Functionality  Identified  and  Implemented  as  a  Result  of 
Field  Testing 

The  table  below  shows  additional  work  needed  and  later  completed  by  this  project. 


Additional  Functionality  Requested 

Technical  Solution 

Need  capability  to  find  previous  attempts 
easily 

Add  a  link  for  the  skill  on  the  dashboard. 
Clicking  on  the  link  will  take  the  coordinator 
to  a  page  which  will  display  all  the  attempts  for 
the  skill  for  that  candidate 

Need  functionality  to  import  instructors  too 

Add  an  import  instructor  feature  similar  to  the 
Import  Candidates  feature,  on  the  Manage 

Users  tab. 

Need  to  be  able  to  create  an  "Instructor  list" 
for  signing  Tablet  PCs  in  and  out  during  the 
All  Skills  Examination 

Add  Export  to  Excel  feature  to  the  Instructor 

List  in  the  Manage  Users  tab. 

Need  to  have  the  ability  to  disable  the 
enforcement  of  the  password  policy  for  the 
users. 

Add  a  check  box  on  Add  User  pages  to  disable 
the  enforcement  of  password  policies. 

Need  to  be  able  to  delete  any  candidate’s 
exams. 

Add  a  delete  button  and  functionality  on  the 
Exams  List  page. 

Need  to  be  able  to  generate  an  excel  report 
of  the  exams  not  taken  by  the  candidates 

Add  Export  to  Excel  functionality  to  the 

Exams  Not  Taken  report 

Need  to  able  to  search  Instructors  by  name 

Add  Search  by  Name  functionality  to  the 
Instructor  List  in  the  Manage  Users  tab. 

Need  the  ability  to  search  candidates  by 
name  on  the  Dashboard 

Add  Search  by  Name  functionality  to  the 
Candidate  Dashboard. 

The  Candidate  roster  report  needs  to  show 
the  count  per  attempt  for  each  skill 

Add  the  additionally  required  counts  to  the 
report.  Also  generate  a  separate  report 
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specifically  for  this. 

Column  headers  on  the  Dashboard  need  to 
be  made  more  easy  to  understand 

Change  the  Dashboard  column  headers 

Company  name  needs  to  contain  the 
number  and  year 

Change  the  company  name  on  all  reports 

Spinal  Immobilization  Supine  Patient  needs 
to  have  "LSB"  suffixed  to  it 

Change  the  name  to  contain  LSB  wherever 
applicable 

Spinal  Immobilization  Seated  Patient  needs 
to  have  "KED"  suffixed  to  it 

Change  the  name  to  contain  KED  wherever 
applicable 

"Reteach"  stamp  should  be  changed  to 
"RE-TEST/RE-TACH  " 

Change  the  image  for  the  same 

Instructions  and  Scenarios  need  to  be 
updated 

Update  the  html  files  for  the  scenarios  and 
instruction 

Need  to  be  able  to  overwrite  a  passed  exam 
to  failed 

Change  the  rescore  feature  to  allow  failing  a 
passed  exam  too 

8.3  Final  Testing  January  2010 

Deloitte  conducted  a  train  the  trainer  session  and  performed  field  system  testing  during  a  four 
day  period.  The  following  personnel  were  in  attendance: 

Deloitte  Personnel: 

Mr.  Pawan  Singh 
Mr.  Raman  Singh 
Mr.  Devchand  Chute 
Mr.  Shivaraman  Raghuraman 

DCMT  Personnel: 

Mr.  Marcus  Vigil 
Ms.  Chavez 
Cpt.  Brookins 
DCMT  Instructors 


8.3.1  Training  and  Field  Trial 

The  visit  to  the  client  site  was  scheduled  to  train  a  set  of  end-users  (Instructors/instructors)  and 
run  the  entire  ALL  SKILLS  EXAMINATION  using  the  application  for  four  of  the  eight  teams 
being  tested. 

Training: 

We  set  the  following  training  objectives  for  this  trip: 

•  Train  users  on  the  process  and  data  flow  of  the  solution  including  Tablet  PC,  Console  and 
Portal  application. 

•  Demonstrate  Tablet  PC  application  including  Exam  navigation,  completion,  signing  and 
submission. 
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•  Demonstrate  Console  application  including  Reporting,  Dashboard,  Re-Score  and 
synchronization  functionality. 

•  Demonstrate  Web  Portal  application  including  data  verification,  reporting  and  printing. 

Field  Trial 

We  also  set  the  following  goals  for  the  field  testing: 

•  The  instructors  use  the  tablet  PC  application  instead  of  the  paper  forms  to  conduct  the  ALL 
SKILLS  EXAMINATION  for  all  the  eight  teams  being  tested. 

•  The  coordinator  uses  the  EMT  Console  application  to  track  status  and  coordinate  the  ALL 
SKILLS  EXAMINATION,  instead  of  manually  compiling  the  results  and  generating  reports. 


8.3.2  Environment  and  Set  Up  Day  1  |  January  7,  2010 

We  setup  the  Tablet  PCs  for  use  in  the  ALL  SKILLS  EXAMINATION.  We  charged  about  50 
tablet  PCs  completely.  We  then  pointed  all  tablet  PCs  to  the  training  server  for  the  instructors  to 
play  around  with  and  get  comfortable  with  the  system.  We  trained  about  20  new  instructors  to 
use  the  Tablet  PC  application  in  a  quick  training  session  which  covered  the  following: 
•Demonstrated  Tablet  PC  application  including  form  navigation,  completion,  signing  and 
submission. 

•We  also  answered  questions  about  the  ability  to  write  on  the  tablet  PCs  and  also  if  portable 
keyboards  could  easily  be  used  to  write  comments  on  the  application. 

•The  instructors  were  then  given  a  tablet  PC  each. 

•A  training  database  was  setup  with  dummy  candidates  and  instructors 

•The  instructors  were  allowed  to  complete  forms  and  submit  exams  for  these  dummy  candidates, 
so  as  to  get  a  feel  of  the  exam. 

We  also  trained  two  new  coordinators  to  use  the  Console  application.  We  then  tested  the  wireless 
network  strength  and  stability  by  walking  around  the  building  and  submitting  forms  from  various 
parts  of  the  building.  The  wireless  environment,  the  tablet  PC  application  and  the  Console 
application  were  not  changed.  We  finally  received  the  list  of  candidates  for  the  exam.  We  trained 
the  coordinators  on  the  import  process  and  also  created  a  template  for  the  candidate  list  that 
could  easily  be  imported  into  the  system.  We  also  added  the  new  instructors  into  the  system. 


8.3.3  Day  2  Field  Trial  Instructors  and  Evaluators  |  January  8,  2010 

Field  Trial  -Instructors 

The  instructors  were  assigned  one  Tablet  PC  each  and  were  signed  out  to  each.  The  candidates 
were  listed  in  the  Tablet  PC  application  and  the  evaluators  were  able  to  pick  them  out  easily.  The 
instructors  used  the  various  search  features  to  select  the  candidate,  and  a  list  of  exams  available 
for  each  candidate  was  shown.  The  instructors  loaded  the  skill  sheet  for  the  required  exam  and 
graded  the  candidate  by  starting  the  timer.  The  instructors  were  also  able  to  read  out  instructions 
to  the  candidates  by  bringing  up  the  instructions  on  the  application.  The  instructors  were  also 
able  to  bring  up  scenarios,  when  necessary,  on  the  Tablet  PC  using  the  application. 


Deloitte-Federal  Healthcare 


49  of  57 


Final  Report  /  Concept  of  Operations  ( CONOPS ) 


DCMT  TE  W81 XWH-07-C-01 1 5 


There  were  about  1686  forms  submitted,  amounting  to  about  three  and  a  half  skills,  by  the  end  of 
the  day. 

Field  Trial  -Coordinators 

The  coordinator  was  able  to  track  exams  as  they  were  being  submitted.  The  coordinator  was  able 
to  track  candidates  who  failed  exams  and  was  able  to  counsel  them  immediately  and  send  them 
to  a  Re-teach  session,  if  necessary.  The  coordinator  was  also  able  to  track  if  any  candidate  failed 
the  ALL  SKILLS  EXAMINATION,  on  an  immediate  basis  and  the  coordinator  was  also  able  to 
generate  reports  as  necessary. 


8.3.4  Day  3  Feedback  and  Results  |  January  11,  2010 

A  total  of  2571  exams  were  submitted  by  the  end  of  the  All  Skills  Examination.  The  coordinator 
only  had  to  except  by  a  click  to  generate  the  required  reports  at  the  end  of  the  examination. 

Instructors  found  the  Tablet  PC  application  much  easier  and  faster  to  use  vs.  the  paper  based 
forms.  Instructors  noticed  the  time  saved  when  the  Tablet  PC  pulled  up  the  forms  with  all  the 
candidate  information  pre-populated. 

Instructors  felt  comfortable  writing  comments  on  the  Tablet  PC,  and  Instructors  liked  the  auto- 
populated  comments  for  critical  criteria,  which  reduced  the  amount  of  redundant  typing  required 
on  their  part.  Instructors  also  noticed  that  the  application  and  wireless  environment  were  very 
stable  with  absolutely  no  issues  throughout  All  Skills  test.  Instructors  noticed  they  saved  time  in 
not  having  to  count  and  grade  the  exam  manually,  as  the  application  did  that  for  them 
automatically. 

Battery  life  of  Tablet  PC  was  down  to  anywhere  from  65%  to  80%  after  3  hours  of  usage. 
Instructors  and  coordinators  pointed  out  the  following  issues  to  us: 

-Need  for  more  generic  comments  for  situations  when  the  candidates  failed  because  of  not 
meeting  the  minimum  score. 

-Need  to  reduce  the  typos  in  the  instructions  and  the  scenarios. 

Overall  the  feedback  was  very  positive. 

8.3.5  Additional  Functionality  implemented  as  a  result  of  the  trip 


Additional  Functionality  Requested 

Technical  Solution 

Need  to  be  able  to  track  class  name  for  each 
candidate 

Add  and  additional  column  called  Class  to  the 
Candidate  Table 

Need  ability  to  track  which  candidates  have 
been  called  for  counseling 

Mark  the  records  that  have  been  printed  on  the 
list  of  exams 

Need  ability  to  filter  the  list  of  failed  exams 
by  instructor 

Add  a  filter  to  the  list  of  failed  exams  to  filter 
by  exams 
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Need  ability  to  be  able  to  repair  the  wireless 
connect  from  within  the  application 

Place  button  into  the  application  or  re-start  the 
router  manually 

Instructors  must  be  forced  to  read 
instructions  to  the  candidate  before 
beginning  the  evaluation 

Lock  the  exam  from  until  the  instructions  to 
the  candidate  page  is  not  opened  by  the 
instructor 

Critical  criteria  should  be  automatically 
checked  based  on  responses  to 
corresponding  questions 

Check  the  responses  to  questions  and  if  it 
matches  requirements,  check  the  corresponding 
critical  criteria  automatically 

Need  for  the  ability  to  fail  the  candidate  if 
time  has  been  exceeded 

Add  a  checkbox  in  the  critical  section  for  time 
exceeded  criteria.  Check  box  should  not 
appear  elsewhere 

Field  Size  for  username  and  password  fields 
need  to  be  bigger  for  easier  access 

Increase  the  size  of  the  text  boxes  required  for 
input 

Instructors  not  aware  of  minimum  score  for 
each  skill 

Comments  need  to  be  automatically  added  to 
skill  sheet  if  the  candidate  does  not  achieve  the 
minimum  score 

9  Conclusion 

9. 1  Strategic  Outcomes 

The  Deloitte  approach  to  this  research  effort  was  straightforward  with  maximum  flexibility 


toward  creation  of  a  useable  end  product. 

Begin  with  a  broad  hypothesis 

The  project  was  conducted  in  to  one-year 
phases.  During  the  first  phase,  we  developed 
the  technology  and  the  initial  requirements 
around  the  hypothesis..  During  the  second 
phase,  we  implemented  the  requirements  into 
the  DCMT  system 

Develop  initial  set  of  requirements  based 
upon  the  hypothesis 

During  the  second  phase,  we  implemented  and 
tested  the  initial  requirements  by  building  the 
DCMT  system 

Work  with  DCMT  personnel  to  review  the 
initial  requirements 

DCMT  personnel  reviewed  and  approved  the 
resulting  set  of  functionalities  provided  by  the 
system 

Develop  the  final  functionality  based  upon 
field  testing  and  adoption  of  enhancements 

During  separate  field  testing  trips,  we  fully 
tested  the  functionality  and  made  many 
enhancements.  DCMT  are  continuing  to  use 
the  system  to  enhance  their  testing  processes. 

What  the  strategy  was  going  in? _ How  did  the  strategy  work? 
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9.2  Performance  Results 

The  table  below  describes  the  results  mapped  to  their  original  hypothesis 


Category 

Criteria 

Results  Achieved  in  the  Pilot 

1.  Increased 

Efficiency 

a.  reduced  paper  use  and 
associated  storage  and 
management  cost 

b.  free  up  manpower  used  as 
runners  and  processors 

c.  instant  scoring/feedback 
including  details  for  corrective 
assistance 

d.  decreased  time  of  overall 
testing  process 

a.  Significant  (eliminates 
paper  involved  in  10,000+ 
individual  exams/year) 

b.  Significant  (eliminated  10 
runners  +  9  processors  per 
shift,  2  per  day,  2  days 
every  other  month  -  1920 
man-hour  s/yr.) 

c.  Significant  (the  ability  for 
instant  scoring  provided 
more  effective  training 
environment  allowing  the 
instructor  instant  access  to 
candidate  achievements 

d.  Significant  (30%  reduction 
in  time  required  for  testing  - 
from  20  hours  to  14  hours) 

2.  Increased 
Accuracy 

a.  legible  capture  of  instructor 
comments 

b.  validation  of  input  prior  to 
submission 

c.  import  of  student  demographics 

a.  Significant 

b.  Significant 

c.  Significant  (instructor 
comments  very  positive  on 
this  being  a  substantial 
time-saver) 

3.  Administrative 
Simplification 

a.  enable  automated  management 
of  performance  indicators 

b.  web-enabled  access 

c.  automated  synchronization  of 
TEMTPC  and  EMT  Web  Portal 

d.  digital  data  sharing  capability 

a.  Significant  (includes 
performance  of  instructors) 

b.  This  feature  declined 

c.  Significant 

d.  Significant  (decreased 
processing  time  and 
shipping/handling  cost) 

4.  Enhanced  Data 
Security 

a.  encryption  during  testing 
process  during  transfer  between 
instructor  to  evaluator 

b.  encrypted  storage 

c.  encryption  for  transfer 

a.  Significant 

b.  Significant 

c.  Significant 
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9.3  General  Comments 

This  engagement  demonstrates  our  ability  to  lead  a  multifaceted,  large-scale  business  process 
and  technology  implementation  in  a  highly  controlled  environment.  Deloitte  was  able  to 
assemble  a  team  with  the  necessary  technology  experience  as  well  as  the  detailed  understanding 
of  DCMT  processes  and  policies  necessary  to  succeed  on  this  project.  We’ve  applied  leading- 
edge  enterprise-service  wireless  technology  to  provide  an  integrated  environment  for  the  client 
without  a  full-scale  replacement  of  its  technology  portfolio.  The  technology,  training  and 
solution  deployment  experience  gained  on  this  project  will  be  able  to  be  used  to  help  other  health 
care  organizations  implement  more  effective  technology  in  a  way  that  will  meet  their 
requirements. 

9.4  Recommendations  to  Expand  Utilizations  into  Other  Areas 

From  a  high  level,  it  was  obvious  that  the  application  and  devices  were  well  received  by  the 
DCMT  users.  Our  intention  was  to  meet  the  overall  project  goals  and  objectives  with  an 
innovative  product.  Inherent  in  any  research  and  proof  of  concept  roll-out  are  both  the 
limitations  of  the  solution  and  opportunities  for  expansion.  At  this  point  in  the  operationalization 
of  the  tools,  the  overall  system  has  moved  into  sustainment.  Recommendations  are  provided 
below: 

9.4.1  Operational  Recommendations 

Operational  recommendations  are  categorized  by  the  following  three  components 

a.  Hardware  -  The  Tablet  PCs  and  Servers  will  need  to  be  maintained  both  from  a  systems 
operation  point  to  keep  the  infrastructure  current  with  latest  patches  as  well  as  physical 
security  of  the  devices  and  replacement  when  damaged  or  defective. 

b.  Software  -  The  applications  loaded  onto  the  laptop  could  become  unstable  and  need  to  be 
re-imaged.  Data  stored  on  the  servers  will  need  to  be  backed  up  with  a  system  restore 
capability  designed  and  tested  to  ensure  that  the  overall  recovery  scheme  will  work. 

c.  Configuration  -  The  system  may  need  to  accommodate  user  generated  requests  for 
additional  functionality  and/or  deletion  of  current  ones. 

9.4.2  DCMT  Expansion 

The  delivered  system  could  be  expanded  with  additional  use  cases  by  adding  functionality  in 
each  of  the  hardware  components 

•  Tablet  PC’s  -  The  current  use  is  limited  to  conducting  the  exams.  These  tools  can  be 
expanded  for  use  as  training  aids  for  students  by  simply  adding  documents,  instruction 
plans  and  commercial  interactive  training  tools  purchased  or  downloaded  from  other 
sources 

•  Servers  -  The  wireless  connections  between  the  server  and  the  PCs  could  also  be  used  for 
group  training  sessions,  surveys,  etc.  as  part  of  an  overall  training  regimen  to  test  while 
training  to  determine  candidate  knowledge  and  retention. 

•  Portal  -  The  portal  could  be  expanded  to  include  such  things  as  content  management, 
additional  reporting,  expanded  business  intelligence,  and  performance  management.  The 
portal  could  also  be  expanded  as  a  recruiting  tool  and  overall  knowledge  management 
portal  to  organize  DCMT  materials. 
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9.4.3  Replication  to  Other  Services 

The  system  could  be  ported  to  other  services  by  minor  changes  to  the  functionality  to 
accommodate  individual  service  needs  or  other  services  could  send  candidates  to  DCMT  for 
training  and  certification.  Either  way,  there  could  be  significant  potential  savings. 
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KEY  RESEARCH  ACCOMPLISHMENTS 


Automating  the  testing  process  improved  the  efficiency  of  the  Practical  Exam  administration 

process  and  enhanced  training  effectiveness  through  adaptive  testing  and  real  time  feedback  on 

performance. 

Piloting  the  automated  process: 

•  Decreased  the  amount  of  time  required  for  scoring  the  practical  exam  and  provides  real-time 
feedback  to  students  and  instructors 

•  Allowed  trend  analysis  of  the  results  of  the  exam  to  identify  training  deficiencies  of  the 
individual  as  well  as  the  program. 

•  Provided  students  additional  learning  opportunities  where  they  are  needed  by  decreasing  the 
overall  amount  of  time  required  for  the  PE. 

•  Supported  the  development  of  a  common  framework  that  will  automate  additional  training 
programs  and  examinations. 

•  Enabled  a  plan  for  enculturation  of  instructors  into  an  environment  of  digitized  training. 

•  Returned  soldiers  currently  used  for  runners  and  data  entry  to  productive  work  and  study. 

•  Identified  opportunities  for  improvements  and  workflow  effectiveness  with  the  use 
integrated  technology 


REPORTABLE  OUTCOMES 

This  project  has  been  used  by  Deloitte  as  a  qualification  to  show  experience  in  the  development 
of  electronic  forms  and/or  secure  wireless  portal  solutions  for  two  winning  engagements  to  date. 


CONCLUSION 

The  reaction  of  DCMT  staff  to  the  pilot  implementation  of  the  prototype  designed  indicates  a 
successful  outcome  in  improving  the  testing  process.  The  coordinator  had  very  little  work 
except  to  click  buttons  to  generate  the  required  reports  at  the  end  of  the  examination.  Instructors 
found  the  Tablet  PC  application  much  easier  and  faster  to  use  vs.  the  paper  based  forms. 
Instructors  noticed  the  time  saved  in  pulling  up  the  forms  on  Tablet  PC  with  all  the  candidate 
information  pre-populated.  Instructors  felt  comfortable  writing  comments  on  the  Tablet  PC. 
Instructors  also  liked  the  auto-populated  comments  for  critical  criteria,  which  reduced  the 
amount  of  redundant  typing  required  on  their  part. 

The  measured  results  also  support  a  successful  study  -  proving  gains  in  efficiency  at  a  level  that 
supports  serious  consideration  for  expanded  use  of  the  technology  used  for  this  study. 
Specifically,  a  reduction  in  the  time  required  for  the  overall  testing  process  of  30  percent  is  a 
major  improvement.  The  reduction  of  man-hours  required  measured  (more  than  1900/year)  did 
not  include  the  time  freed  up  by  the  reduction  in  the  time  for  the  testing  process,  it  is  simply  a 
reduction  in  needed  manpower  for  runners  and  processing  personnel  -  the  reduction  of  overall 
testing  time  allowed  the  staff  to  reduce  overtime  demands  for  the  first  day  of  testing. 

The  application  of  technology  to  automate  paper-based  processes  is  often  thought  of  as 
expensive  due  to  the  initial  investment,  however  the  combination  of  efficiencies  gained  and  the 
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improved  capability  for  managing  data  that  extends  beyond  the  reach  of  the  original  paper-based 
process  is  important  to  consider  as  well.  The  pilot  allows  the  course  coordinator  to  evaluate  not 
only  students  taking  the  exam,  but  also  to  do  analysis  on  test  stations  and  instructors  that  will 
provide  insight  for  improving  teaching  methods  and  have  a  definitive  impact  on  the  overall 
outcome  of  the  training  program. 

The  testing  and  evaluation  process  for  the  pilot  system  provided  numerous  opportunities  for  the 
staff  and  the  Deloitte  team  to  interact,  identify  potential  improvements,  take  action  on  those 
inputs,  and  resulted  in  a  final  product  that  continues  to  be  lauded  by  the  68W  training  program 
staff. 
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10  Appendix  A 


CKMBT_ 6000901271 
45101Exemption.pdf 
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